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Description 

BACKGROUND OF THE INVENTION 

5 [0001] The present invention relates to a system for managing audiovisual information, and in particular to a system 
for audiovisual information browsing, filtering, searching, archiving, and personalization. 

[0002] Video cassette recorders (VCRs) may record video programs in response to pressing a record button or may 
be programmed to record video programs based on the time of day. However, the viewer must program the VCR based 
on information from a television guide to identify relevant programs to record. After recording, the viewer scans through 

10 the entire video tape to select relevant portions of the program for viewing using the functionality provided by the VCR, 
such as fast forward and fast reverse. Unfortunately, the searching and viewing is based on a linear search, which may 
require significant time to locate the desired portions of the program(s) and fast forward to the desired portion of the 
tape. In addition, it is time consuming to program the VCR in light of the television guide to record desired prog, aims. 
Also, unless the viewer recognizes the programs from the television guide as desirable it is unlikely that the viewer will 

15 select such programs to be recorded. 

[0003] RePlayTV and TiVo have developed hard disk based systems that receive, record, and play television broad- 
casts in a manner similar to a VCR. The systems may be programmed with the viewer's viewing preferences. The 
systems use a telephone line interface to receive scheduling information similar to that available from a television 
guide. Based upon the system programming and the scheduling information, the system automatically records pro- 

20 grams that may be of potential interest to the viewer. Unfortunately, viewing the recorded programs occurs in a linear 
manner and may require substantial time. In addition, each system must be programmed for an individual's preference, 
likely in a different manner. 

[0004] Freeman et al., U.S. Patent No. 5,861 ,881 , disclose an interactive computer system where subscribers can 
receive individualized content. 

25 [0005] With all the aforementioned systems, each individual viewer is required to program the device according to 
his particular viewing preferences. Unfortunately, each different type of device has different capabilities and limitations 
which limit the selections of the viewer. In addition, each device includes a different interface which the viewer may be 
unfamiliar with. Further, if the operator's manual is inadvertently misplaced it may be difficult for the viewer to efficiently 
program the device. 

30 

BRIEF SUMMARY OF THE INVENTION 

[0006] Accordingly, a primary object of the present invention is to provide a system with at least one of audio, image, 
and a video comprising a plurality of frames comprising the step of providing a preferences description (500), describing 
35 preferences of a user with respect to the use of said at least one of said audio, image, and video, where said description 
is a description about a recording quality at the time of recording at least one of said audio, image, and video on a 
storage means. 

[0007] Another object of the system with at least one of audio, image, and a video comprising a plurality of frames 
comprising the step of providing a preferences description (500), describing preferences of a user with respect to the 
■*o use of said at least one of said audio, image, and video, where said description is a description about at least one of 
safeguard time intervals before a program start time and after a program end time at the time of recording at least one 
of said audio, image, and video on a storage means. 

[0008] Another object of the system with at least one of audio, image, and a video comprising a plurality of frames 
comprising the step of providing a preferences description (500), describing preferences of a user with respect to the 
■*5 use of said at least one of said audio, image, and video, where said description is a description about a playback 
preferences (543) at the time of playing back at least one of said audio, image, and video. 

[0009] Another object of the system with at least one of audio, image, and a video comprising a plurality of frames 
comprising the step of providing a preferences description (500), describing preferences of a user with respect to the 
use of said at least one of said audio, image, and video, where said description is a description about a creation date 

50 (547) for at least one content of said audio, image, and video. 

[0010] A further object of the system with at least one of audio, image, and a video comprising a plurality of frames 
comprising the step of providing a preferences description (500), describing preferences of a user with respect to the 
use of said at least one of said audio, image, and video, where said description is a description about a file format 
(545) at the time of recording at least one of said audio, image, and video on a storage means. 

55 [001 1] The foregoing and other objectives, features and advantages of the invention will be more readily understood 
upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying 
drawings. 
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BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 

[0012] FIG. 1 is an exemplary embodiment of a program, a system, and a user, with associated description schemes, 
of an audiovisual system of the present invention. 

[0013] FIG. 2 is an exemplary embodiment of the audiovisual system, including an analysis module, of FIG. 1 . 

[0014] FIG. 3 is an exemplary embodiment of the analysis module of FIG. 2. 

[0015] FIG. 4 is an illustration of a thumbnail view (category) for the audiovisual system. 

[0016] FIG. 5 is an illustration of a thumbnail view (channel) for the audiovisual system. 

[0017] FIG. 6 is an illustration of a text view (channel) for the audiovisual system. 

[0018] FIG. 7 is an illustration of a frame view for the audiovisual system. 

[0019] FIG. 8 is an illustration of a shot view for the audiovisual system. 

[0020] FIG. 9 is an illustration of a key frame view the audiovisual system. 

[0021] FIG. 10 is an illustration of a highlight view for the audiovisual system. 

[0022] FIG. 1 1 is an illustration of an event view for the audiovisual system. 

[0023] FIG. 1 2 is an illustration of a character/object view for the audiovisual system. 

[0024] FIG. 13 is an alternative embodiment of a program description scheme including a syntactic structure de- 
scription scheme, a semantic structure description scheme, a visualization description scheme, and a meta information 
description scheme. 

[0025] FIG. 14 is an exemplary embodiment of the visualization description scheme of FIG. 13. 

[0026] FIG. 15 is an exemplary embodiment of the meta information description scheme of FIG. 13. 

[0027] FIG. 1 6 is an exemplary embodiment of a segment description scheme for the syntactic structure description 

scheme of FIG. 13. 

[0028] FIG. 17 is an exemplary embodiment of a region description scheme for the syntactic structure description 
scheme of FIG. 13. 

[0029] FIG. 1 8 is an exemplary embodiment of a segment/region relation description scheme for the syntactic struc- 
ture description scheme of FIG. 13. 

[0030] FIG. 1 9 is an exemplary embodiment of an event description scheme for the semantic structure description 
scheme of FIG. 13. 

[0031] FIG. 20 is an exemplary embodiment of an object description scheme for the semantic structure description 
scheme of FIG. 13. 

[0032] FIG. 21 is an exemplary embodiment of an event/object relation graph description scheme for the syntactic 
structure description scheme of FIG. 13. 

[0033] FIG. 22 is an exemplary embodiment of a user preference description scheme. 

[0034] FIG. 23 is an exemplary embodiment of the interrelationship between a usage history description scheme, 
an agent, and the usage preference description scheme of FIG. 22. 

[0035] FIG. 24 is an exemplary embodiment of the interrelationship between audio and/or video programs together 

with their descriptors, user identification, and the usage preference description scheme of FIG. 22. 

[0036] FIG. 25 is an exemplary embodiment of a usage preference description scheme of FIG. 22. 

[0037] FIG. 26 is an exemplary embodiment of the interrelationship between the usage description schemes and an 

MPEG-7 description schemes. 

[0038] FIG. 27 is an exemplary embodiment of a usage history description scheme of FIG. 22. 
[0039] FIG. 28 is an exemplary system incorporating the user history description scheme. 
[0040] FIG. 29 is an exemplary user preferences description scheme. 
[0041] FIG. 30 is an exemplary user preferences description scheme. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

[0042] Many households today have many sources of audio and video information, such as multiple television sets, 
multiple VCR's, a home stereo, a home entertainment center, cable television, satellite television, internet broadcasts, 
world wide web, data services, specialized Internet services, portable radio devices, and a stereo in each of their 
vehicles. For each of these devices, a different interface is normally used to obtain, select, record, and play the video 
and/or audio content. For example, a VCR permits the selection of the recording times but the user has to correlate 
the television guide with the desired recording times. Another example is the user selecting a preferred set of prese- 
lected radio stations for his home stereo and also presumably selecting the same set of preselected stations for each 
of the user's vehicles. If another household member desires a different set of preselected stereo selections, the pro- 
gramming of each audio device would need to be reprogrammed at substantial inconvenience. 
[0043] The present inventors came to the realization that users of visual information and listeners to audio information, 
such as for example radio, audio tapes, video tapes, movies, and news, desire to be entertained and informed in more 
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than merely one uniform manner. In other words, the audiovisual information presented to a particular user should be 
in a format and include content suited to their particular viewing preferences. In addition, the format should be dependent 
on the content of the particular audiovisual information. The amount of information presented to a user or a listener 
should be limited to only the amount of detail desired by the particular user at the particular time. For example with the 

5 ever increasing demands on the user's time, the user may desire to watch only 10 minutes of or merely the highlights 
of a basketball game. In addition, the present inventors came to the realization that the necessity of programming 
multiple audio and visual devices with their particular viewing preferences is a burdensome task, especially when 
presented with unfamiliar recording devices when traveling. When traveling, users desire to easily configure unfamiliar 
devices, such as audiovisual devices in a hotel room, with their viewing and listening preferences in a efficient manner. 

w [0044] The present inventors came to the further realization that a convenient technique of merely recording the 
desired audio and video information is not sufficient because the presentation of the information should be in a manner 
that is time efficient, especially in light of the limited time frequently available for the presentation of such information. 
In addition, the user should be able to access only that portion of all of the available information that the user is interested 
in, while skipping the remainder of the information. 

15 [0045] A user is not capable of watching or otherwise listening to the vast potential amount of information available 
through all, or even a small portion of, the sources of audio and video information. In addition, with the increasing 
information potentially available, the user is not likely even aware of the potential content of information that he may 
be interested in. In light of the vast amount of audio, image, and video information, the present inventors came to the 
realization that a system that records and presents to the user audio and video information based upon the user's prior 

20 viewing and listening habits, preferences, and personal characteristics, generally referred to as user information, is 
desirable. In addition, the system may present such information based on the capabilities of the system devices. This 
permits the system to record desirable information and to customize itself automatically to the user and/or listener. It 
is to be understood that user, viewer, and/or listener terms may be used interchangeability for any type of content. 
Also, the user information should be portable between and usable by different devices so that other devices may 

25 likewise be configured automatically to the particular user's preferences upon receiving the viewing information. 

[0046] In light of the foregoing realizations and motivations, the present inventors analyzed a typical audio and video 
presentation environment to determine the significant portions of the typical audiovisual environment. First, referring 
to FIG. 1 the video, image, and/or audio information 10 is provided or otherwise made available to a user and/or a 
(device) system. Second, the video, image, and/or audio information is presented to the user from the system 12 

30 (device), such as a television set or a radio. Third, the user interacts both with the system (device) 12 to view the 
information 10 in a desirable manner and has preferences to define which audio, image, and/or video information is 
obtained in accordance with the user information 1 4. After the proper identification of the different major aspects of an 
audiovisual system the present inventors then realized that information is needed to describe the informational content 
of each portion of the audiovisual system 16. 

35 [0047] With three portions of the audiovisual presentation system 16 identified, the functionality of each portion is 
identified together with its interrelationship to the other portions. To define the necessary interrelationships, a set of 
description schemes containing data describing each portion is defined. The description schemes include data that is 
auxiliary to the programs 10, the system 12, and the user 14, to store a set of information, ranging from human readable 
text to encoded data, that can be used in enabling browsing, filtering, searching, archiving, and personalization. By 

40 providing a separate description scheme describing the program(s) 10, the user 14, and the system 12, the three 
portions (program, user, and system) may be combined together to provide an interactivity not previously achievable. 
In addition, different programs 10, different users 14, and different systems 12 may be combined together in any com- 
bination, while still maintaining full compatibility and functionality. It is to be understood that the description scheme 
may contain the data itself or include links to the data, as desired. 

45 [0048] A program description scheme 18 related to the video, still image, and/or audio information 10 preferably 
includes two sets of information, namely, program views and program profiles. The program views define logical struc- 
tures of the frames of a video that define how the video frames are potentially to be viewed suitable for efficient browsing. 
For example the program views may contain a set of fields that contain data for the identification of key frames, segment 
definitions between shots, highlight definitions, video summary definitions, different lengths of highlights, thumbnail 

so set of frames, individual shots or scenes, representative frame of the video, grouping of different events, and a close- 
up view. The program view descriptions may contain thumbnail, slide, keyframe, highlights, and close-up views so 
that users can filter and search not only at the program level but also within a particular program. The description 
scheme also enables users to access information in varying detail amounts by supporting, for example, a key frame 
view as apart of a program view providing multiple levels of summary ranging from coarse to fine. The program profiles 

55 define distinctive characteristics of the content of the program, such as actors, stars, rating, director, release date, time 
stamps, keyword identification, trigger profile, still profile, event profile, character profile, object profile, color profile, 
texture profile, shape profile, motion profile, and categories. The program profiles are especially suitable to facilitate 
filtering and searching of the audio and video information. The description scheme enables users to have the provision 
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of discovering interesting programs that they may be unaware of by providing a user description scheme. The user 
description scheme provides information to a software agent that in turn performs a search and filtering on behalf of 
the user by possibly using the system description scheme and the program description scheme information. It is to be 
understood that in one of the embodiments of the invention merely the program description scheme is included. 
5 [0049] Program views contained in the program description scheme are a feature that supports a functionality such 
as close-up view. In the close-up view, a certain image object, e.g. , a famous basketball player such as Michael Jordan, 
can be viewed up close by playing back a close-up sequence that is separate from the original program. An alternative 
view can be incorporated in a straightforward manner. Character profile on the other hand may contain spatio-temporal 
position and size of a rectangular region around the character of interest. This region can be enlarged by the presen- 
ce tation engine, or the presentation engine may darken outside the region to focus the user's attention to the characters 
spanning a certain number of frames. Information within the program description scheme may contain data about the 
initial size or location of the region, movement of the region from one frame to another, and duration and terms of the 
dumber of frames featuring the region. The character profile also provides provision for including text annotation and 
audio annotation about the character as well as web page information, and any other suitable information. Such char- 
's acter profiles may include the audio annotation which is separate from and in addition to the associated audio track of 
the video. 

[0050] The program description scheme may likewise contain similar information regarding audio (such as radio 
broadcasts) and images (such as analog or digital photographs or a frame of a video). 

[0051] The user description scheme 20 preferably includes the user's personal preferences, and information regard- 

20 ing the user's viewing history such as for example browsing history, filtering history, searching history, and device 
setting history. The user's personal preferences includes information regarding particular programs and categorizations 
of programs that the user prefers to view. The user description scheme may also include personal information about 
the particular user, such as demographic and geographic information, e.g. zip code and age. The explicit definition of 
the particular programs or attributes related thereto permits the system 1 6 to select those programs from the information 

25 contained within the available program description schemes 1 8 that may be of interest to the user. Frequently, the user 
does not desire to learn to program the device nor desire to explicitly program the device. In addition , the user description 
scheme 20 may not be sufficiently robust to include explicit definitions describing all desirable programs for a particular 
user. In such a case, the capability of the user description scheme 20 to adapt to the viewing habits of the user to 
accommodate different viewing characteristics not explicitly provided for or otherwise difficult to describe is useful. In 

30 such a case, the user description scheme 20 may be augmented or any technique can be used to compare the infor- 
mation contained in the user description scheme 20 to the available information contained in the program description 
scheme 1 8 to make selections . The user description scheme provides a technique for holding user preferences ranging 
from program categories to program views, as well as usage history. User description scheme information is persistent 
but can be updated by the user or by an intelligent software agent on behalf of the user at any arbitrary time. It may 

35 also be disabled by the user, at any time, if the user decides to do so. In addition, the user description scheme is 
modular and portable so that users can carry or port it from one device to another, such as with a handheld electronic 
device or smart card or transported over a network connecting multiple devices. When user description scheme is 
standardized among different manufacturers or products, user preferences become portable. For example, a user can 
personalize the television receiver in a hotel room permitting users to access information they prefer at any time and 

40 anywhere. In a sense, the user description scheme is persistent and timeless based. In addition, selected information 
within the program description scheme may be encrypted since at least part of the information may be deemed to be 
private (e.g., demographics). A user description scheme may be associated with an audiovisual program broadcast 
and compared with a particular user's description scheme of the receiver to readily determine whether or not the 
program's intended audience profile matches that of the user. It is to be understood that in one of the embodiments of 

45 the invention merely the user description scheme is included. 

[0052] The system description scheme 22 preferably manages the individual programs and other data. The man- 
agement may include maintaining lists of programs, categories, channels, users, videos, audio, and images. The man- 
agement may include the capabilities of a device for providing the audio, video, and/or images. Such capabilities may 
include, for example, screen size, stereo, AC3, DTS, color, black/white, etc. The management may also include rela- 

50 tionships between any one or more of the user, the audio, and the images in relation to one or more of a program 
description scheme(s) and a user description scheme(s). In a similar manner the management may include relation- 
ships between one or more of the program description scheme(s) and user description scheme(s). It is to be understood 
that in one of the embodiments of the invention merely the system description scheme is included. 
[0053] The descriptors of the program description scheme and the user description scheme should overlap, at least 

55 partially, so that potential desirability of the program can be determined by comparing descriptors representative of the 
same information. For example, the program and user description scheme may include the same set of categories and 
actors. The program description scheme has no knowledge of the user description scheme, and vice versa, so that 
each description scheme is not dependant on theotherfor its existence. It is not necessary for the description schemes 
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to be fully populated. It is also beneficial not to include the program description scheme with the user description 
scheme because there will likely be thousands of programs with associated description schemes which if combined 
with the user description scheme would result in a unnecessarily large user description scheme. It is desirable to 
maintain the user description scheme small so that it is more readily portable. Accordingly, a system including only the 
program description scheme and the user description scheme would be beneficial. 

[0054] The user description scheme and the system description scheme should include at least partially overlapping 
fields. With overlapping fields the system can capture the desired information, which would otherwise not be recognized 
as desirable. The system description scheme preferably includes a list of users and available programs. Based on the 
master list of available programs, and associated program description scheme, the system can match the desired 
programs. It is also beneficial not to include the system description scheme with the user description scheme because 
there will likely be thousands of programs stored in the system description schemes which if combined with the user 
description scheme would result in a unnecessarily large user description scheme. It is desirable to maintain the user 
description scheme small so that it is more readily portable. For example, the user description scheme may include 
radio station preselected frequencies and/or types of stations, while the system description scheme includes the avail- 
able stations for radio stations in particular cities . When traveling to a different city the user description scheme together 
with the system description scheme will permit reprogramming the radio stations. Accordingly, a system including only 
the system description scheme and the user description scheme would be beneficial. 

[0055] The program description scheme and the system description scheme should include at least partially over- 
lapping fields. With the overlapping fields, the system description scheme will be capable of storing the information 
contained within the program description scheme, so that the information is properly indexed. With proper indexing, 
the system is capable of matching such information with the user information, if available, for obtaining and recording 
suitable programs. If the program description scheme and the system description scheme were not overlapping then 
no information would be extracted from the programs and stored. System capabilities specified within the system 
description scheme of a particular viewing system can be correlated with a program description scheme to determine 
the views that can be supported by the viewing system. For instance, if the viewing device is not capable of playing 
back video, its system description scheme may describe its viewing capabilities as limited to keyframe view and slide 
view only. Program description scheme of a particular program and system description scheme of the viewing system 
are utilized to present the appropriate views to the viewing system. Thus, a server of programs serves the appropriate 
views according to a particular viewing system's capabilities, which may be communicated over a network or commu- 
nication channel connecting the server with user's viewing device. It is preferred to maintain the program description 
scheme separate from the system description scheme because the content providers repackage the content and de- 
scription schemes in different styles, times, and formats. Preferably, the program description scheme is associated 
with the program, even if displayed at a different time. Accordingly, a system including only the system description 
scheme and the program description scheme would be beneficial. 

[0056] By preferably maintaining the independence of each of the three description schemes while having fields that 
correlate the same information, the programs 10, the users 14, and the system 12 may be interchanged with one 
another while maintaining the functionality of the entire system 16. Referring to FIG. 2, the audio, visual, or audiovisual 
program 38, is received by the system 16. The program 38 may originate at any suitable source, such as for example 
broadcast television, cable television, satellite television, digital television, Internet broadcasts, world wide web, digital 
video discs, still images, video cameras, laser discs, magnetic media, computer hard drive, video tape, audio tape, 
data services, radio broadcasts, and microwave communications. The program description stream may originate from 
any suitable source, such as for example PS IP/DVB-SI information in digital television broadcasts, specialized digital 
television data services, specialized Internet services, world wide web, data files, data over the telephone, and memory, 
such as computer memory. The program, user, and/or system description scheme may be transported over a network 
(communication channel). For example, the system description scheme may be transported to the source to provide 
the source with views or other capabilities that the device is capable of using. In response, the source provides the 
device with image, audio, and/or video content customized or otherwise suitable for the particular device. The system 
1 6 may include any device(s) suitable to receive any one or more of such programs 38. An audiovisual program analysis 
module 42 performs an analysis of the received programs 38 to extract and provide program related information (de- 
scriptors) to the description scheme (DS) generation module 44. The program related information may be extracted 
from the data stream including the program 38 or obtained from any other source, such as for example data transferred 
over a telephone line, data already transferred to the system 1 6 in the past, or data from an associated file. The program 
related information preferably includes data defining both the program views and the program profiles available for the 
particular program 38. The analysis module 42 performs an analysis of the programs 38 using information obtained 
from (i) automatic audio-video analysis methods on the basis of low-level features that are extracted from the program 
(s), (ii) event detection techniques, (iii) data that is available (or extractable) from data sources or electronic program 
guides (EPGs, DVB-SI, and PSIP), and (iv) user information obtained from the user description scheme 20 to provide 
data defining the program description scheme. 
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[0057] The selection of a particular program analysis technique depends on the amount of readily available data and 
the user preferences. For example, if a user prefers to watch a 5 minute video highlight of a particular program, such 
as a basketball game, the analysis module 42 may invoke a knowledge based system 90 (FIG. 3) to determine the 
highlights that form the best 5 minute summary. The knowledge based system 90 may invoke a commercial filter 92 

5 to remove commercials and a slow motion detector 54 to assist in creating the video summary. The analysis module 
42 may also invoke other modules to bring information together (e.g., textual information) to author particular program 
views. For example, if the program 38 is a home video where there is no further information available then the analysis 
module 42 may create a key-frame summary by identifying keyframes of a multi-level summary and passing the infor- 
mation to be used to generate the program views, and in particular a key frame view, to the description scheme. 

10 Referring also to FIG. 3, the analysis module 42 may also include other sub-modules, such as for example, a de-mux/ 
decoder 60, a data and service content analyzer 62, a text processing and text summary generator 64, a close caption 
analyzer 66, a title frame generator 68, an analysis manager 70, an audiovisual analysis and feature extractor 72, an 
evsnt detector 74, a key-frame summarizer 76, and a highlight summarizer78. 

[0058] The generation module 44 receives the system information 46 for the system description scheme. The system 

15 information 46 preferably includes data for the system description scheme 22 generated by the generation module 44. 
The generation module 44 also receives user information 48 including data for the user description scheme. The user 
infoimation 48 preferably includes data for the user description scheme generated within the generation module 44. 
The user input 48 may include, for example, meta information to be included in the program and system description 
scheme. The user description scheme (or corresponding information) is provided to the analysis module 42 for selective 

20 analysis of the program(s) 38. For example, the user description scheme may be suitable for triggering the highlight 
generation functionality for a particular program and thus generating the preferred views and storing associated data 
in the program description scheme. The generation module 44 and the analysis module 42 provide data to a data 
storage unit 50. The storage unit 50 may be any storage device, such as memory or magnetic media. 
[0059] A search, filtering, and browsing (SFB) module 52 implements the description scheme technique by parsing 

25 and extracting information contained within the description scheme. The SFB module 52 may perform filtering, search- 
ing, and browsing of the programs 38, on the basis of the information contained in the description schemes. An intelligent 
software agent is preferably included within the SFB module 52 that gathers and provides user specific information to 
the generation module 44 to be used in authoring and updating the user description scheme (through the generation 
module 44). In this manner, desirable content may be provided to the user though a display 80. The selections of the 

30 desired program(s) to be retrieved, stored, and/or viewed may be programmed, at least in part, through a graphical 
user interface 82. The graphical user interface may also include or be connected to a presentation engine for presenting 
the information to the user through the graphical user interface. 

[0060] The intelligent management and consumption of audiovisual information using the multi-part description 
stream device provides a next-generation device suitable for the modern era of information overload. The device re- 
35 sponds to changing lifestyles of individuals and families, and allows everyone to obtain the information they desire 
anytime and anywhere they want. 

[0061] An example of the use of the device may be as follows. A user comes home from work late Friday evening 
being happy the work week is finally over. The user desires to catch up with the events of the world and then watch 
ABC's 20/20 show later that evening. It is now 9 PM and the 20/20 show will start in an hour at 10 PM. The user is 

40 interested in the sporting events of the week, and all the news about the Microsoft case with the Department of Justice. 
The user description scheme may include a profile indicating a desire that the particular user wants to obtain all available 
information regarding the Microsoft trial and selected sporting events for particular teams. In addition, the system 
description scheme and program description scheme provide information regarding the content of the available infor- 
mation that may selectively be obtained and recorded. The system, in an autonomous manner, periodically obtains 

45 and records the audiovisual information that may be of interest to the user during the past week based on the three 
description schemes. The device most likely has recorded more than one hour of audiovisual information so the infor- 
mation needs to be condensed in some manner. The user starts interacting with the system with a pointer or voice 
commands to indicate a desire to view recorded sporting programs. On the display, the user is presented with a list of 
recorded sporting events including Basketball and Soccer. Apparently the user's favorite Football team did not play 

50 that week because it was not recorded. The user is interested in basketball games and indicates a desire to view 
games. A set of title frames is presented on the display that captures an important moment of each game. The user 
selects the Chicago Bulls game and indicates a desire to view a 5 minute highlight of the game. The system automat- 
ically generates highlights. The highlights may be generated by audio or video analysis, or the program description 
scheme includes data indicating the frames that are presented for a 5 minute highlight. The system may have also 

55 recorded web-based textual information regarding the particular Chicago-Bulls game which may be selected by the 
user for viewing. If desired, the summarized information may be recorded onto a storage device, such as a DVD with 
a label. The stored information may also include an index code so that it can be located at a later time. After viewing 
the sporting events the user may decide to read the news about the Microsoft trial. It is now 9:50 PM and the user is 
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done viewing the news. In fact, the user has selected to delete all the recorded news items after viewing them. The 
user then remembers to do one last thing before 10 PM in the evening. The next day, the user desires to watch the 
VHS tape that he received from his brother that day, containing footage about his brother's new baby girl and his 
vacation to Peru last summer. The user wants to watch the whole 2-hour tape but he is anxious to see what the baby 

5 looks like and also the new stadium built in Lima, which was not there last time he visited Peru. The user plans to take 
a quick look at a visual summary of the tape, browse, and perhaps watch a few segments for a couple of minutes, 
before the user takes his daughter to her piano lesson at 1 0 AM the next morning. The user plugs in the. tape into his 
VCR, that is connected to the system, and invokes the summarization functionality of the system to scan the tape and 
prepare a summary. The user can then view the summary the next morning to quickly discover the baby's looks, and 

10 playback segments between the keyframes of the summary to catch a glimpse of the crying baby. The system may 
also record the tape content onto the system hard drive (or storage device) so the video summary can be viewed 
quickly. It is now 10:10 PM, and it seems that the user is 10 minutes late for viewing 20/20. Fortunately, the system, 
based on the three description schemes, has already been recording 20/20 since 10 PM. Now the user can start 
watching the recorded portion of 20/20 as the recording of 20/20 proceeds. The user will be done viewing 20/20 at 11 : 

15 10 PM. 

[0062] The average consumer has an ever increasing number of multimedia devices, such as a home audio system, 
a car stereo, several home television sets, web browsers, etc. The user currently has to customize each of the devices 
for optimal viewing and/or listening preferences. By storing the user preferences on a removable storage device, such 
as a smart card, the user may insert the card including the user preferences into such media devices for automatic 

20 customization. This results in the desired programs being automatically recorded on the VCR, and setting of the radio 
stations for the car stereo and home audio system. In this manner the user only has to specify his preferences at most 
once, on a single device and subsequently, the descriptors are automatically uploaded into devices by the removable 
storage device. The user description scheme may also be loaded into other devices using a wired or wireless network 
connection, e.g. that of a home network. Alternatively, the system can store the user history and create entries in the 

25 user description scheme based on the's audio and video viewing habits. In this manner, the user would never need to 
program the viewing information to obtain desired information. In a sense, the user descriptorscheme enables modeling 
of the user by providing a central storage for the user's listening, viewing, browsing preferences, and user's behavior. 
This enables devices to be quickly personalized, and enables other components, such as intelligent agents, to com- 
municate on the basis of a standardized description format, and to make smart inferences regarding the user's pref- 

30 erences. 

[0063] Many different realizations and applications can be readily derived from FIGS. 2 and 3 by appropriately or- 
ganizing and utilizing their different parts, or by adding peripherals and extensions as needed. In its most general form, 
FIG. 2 depicts an audiovisual searching, filtering, browsing, and/or recording appliance that is personalizable. The list 
of more specific applications/implementations given below is not exhaustive but covers a range. 

35 [0064] The user description scheme is a major enabler for personalizable audiovisual appliances. If the structure 
(syntax and semantics) of the description schemes is known amongst multiple appliances, the user (user) can carry 
(or otherwise transfer) the information contained within his user description scheme from one appliance to another, 
perhaps via a smart card-where these appliances support smart card interface- in order to personalize them. Per- 
sonalization can range from device settings, such as display contrast and volume control, to settings of television 

40 channels, radio stations, web stations, web sites, geographic information, and demographic information such as age, 
zip code etc. Appliances that can be personalized may access content from different sources. They may be connected 
to the web, terrestrial or cable broadcast, etc., and they may also access multiple or different types of single media 
such as video, music, etc. 

[0065] For example, one can personalize the car stereo using a smart card plugged out of the home system and 
is plugged into the car stereo system to be able to tune to favorite stations at certain times. As another example, one can 
also personalize television viewing, for example, by plugging the smart card into a remote control that in turn will 
autonomously command the television receiving system to present the user information about current and future pro- 
grams that fits the user's preferences. Different members of the household can instantly personalize the viewing ex- 
perience by inserting their own smart card into the family remote. In the absence of such a remote, this same type of 
so personalization can be achieved by plugging in the smart card directly to the television system. The remote may likewise 
control audio systems. In another implementation, the television receiving system holds user description schemes for 
multiple users (users) in local storage and identify different users (or group of users) by using an appropriate input 
interface. For example an interface using user-voice identification technology. It is noted that in a networked system 
the user description scheme may be transported over the network. 
55 [0066] The user description scheme is generated by direct user input, and by using a software that watches the user 
to determine his/her usage pattern and usage history. User description scheme can be updated in a dynamic fashion 
by the user or automatically. A well defined and structured description scheme design allows different devices to in- 
teroperate with each other. A modular design also provides portability. 
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[0067] The description scheme adds new functionality to those of the current VCR. An advanced VCR system can 
learn from the user via direct input of preferences, or by watching the usage pattern and history of the user. The user 
description scheme holds user's preferences users and usage history. An intelligent agent can then consult with the 
user description scheme and obtain information that it needs for acting on behalf of the user. Through the intelligent 
s agent, the system acts on behalf of the user to discover programs that fit the taste of the user, alert the user about 
such programs, and/or record them autonomously. An agent can also manage the storage in the system according to 
the user description scheme, i.e., prioritizing the deletion of programs (or alerting the user for transfer to a removable 
media), or determining their compression factor (which directly impacts their visual quality) according to user's prefer- 
ences and history. 

10 [0068] The program description scheme and the system description scheme work in collaboration with the user 
description scheme in achieving some tasks. In addition, the program description scheme and system description 
scheme in an advanced VCR or other system will enable the user to browse, search, and filter audiovisual programs. 
Browsing in the system offers capabilities that are well beyond fast forwarding and rewinding. For instance, the user 
can view a thumbnail view of different categories of programs stored in the system. The user then may choose frame 

'5 view, shot view, key frame view, or highlight view, depending on their availability and user's preference. These views 
can be readily invoked using the relevant information in the program description scheme, especially in program views. 
The user at any time can start viewing the program either in parts, or in its entirety. 

[0069] In this application, the program description scheme may be readily, available from many services such as: (i) 
from Broadcast (carried by EPG defined as a part of ATSC-PSIP (ATSC-Program Service Integration Protocol) in USA 
20 or DVB-SI (Digital Video Broadcast-Service Information) in Europe); (ii) from specialized data services (in addition to 
PSIP/DVB-SI); (iii) from specialized web sites; (iv) from the media storage unit containing the audiovisual content (e. 
g DVD): (v) from advanced cameras (discussed later), and/or may be generated (i.e., for programs that are being 
stored) by the analysis module 42 or by user input 48. 

[0070] Contents of digital still and video cameras can be stored and managed by a system that implements the 
25 description schemes, e.g., a system as shown in FIG. 2. Advanced cameras can store a program description scheme, 
for instance, in addition to the audiovisual content itself. The program description scheme can be generated either in 
part or in its entirety on the camera itself via an appropriate user input interface (e.g., speech, visual menu drive, etc.). 
Users can input to the camera the program description scheme information, especially those high-level (or semantic) 
information that may otherwise be difficult to automatically extract by the system . Some camera settings and parameters 
30 (e.g.. date and time), as well as quantities computed in the camera (e.g., color histogram to be included in the color 
profile), can also be used in generating the program description scheme. Once the camera is connected, the system 
can browse the camera content, or transfer the camera content and its description scheme to the local storage for 
future use. It is also possible to update or add information to the description scheme generated in the camera. 
[0071] The IEEE 1394 and Havi standard specifications enable this type of "audiovisual content" centric communi- 
35 cation among devices. The description scheme API's can be used in the context of Havi to browse and/or search the 
contents of a camera or a DVD which also contain a description scheme associated with their content, i.e., doing more 
than merely invoking the PLAY API to play back and linearly view the media. 

[0072] The description schemes may be used in archiving audiovisual programs in a database. The search engine 
uses the information contained in the program description scheme to retrieve programs on the basis of their content. 
40 The program description scheme can also be used in navigating through the contents of the database or the query 
results. The user description scheme can be used in prioritizing the results of the user query during presentation. It is 
possible of course to make the program description scheme more comprehensive depending on the nature of the 
particular application. 

[0073] The description scheme fulfills the user's desire to have applications that pay attention and are responsive 
45 to their viewing and usage habits, preferences, and personal demographics. The proposed user description scheme 
directly addresses this desire in its selection of fields and interrelationship to other description schemes. Because the 
description schemes are modular in nature, the user can port his user description scheme from one device to another 
in order to "personalize" the device. 

[0074] The proposed description schemes can be incorporated into current products similar to those from TiVo and 
50 Replay TV in order to extend their entertainment informational value. In particular, the description scheme will enable 
audiovisual browsing and searching of programs and enable filtering within a particular program by supporting multiple 
program views such as the highlight view. In addition, the description scheme will handle programs coming from sources 
other than television broadcasts for which TiVo and Replay TV are not designed to handle, In addition, by standardi- 
zation of TiVo and Replay TV type of devices, other products may be interconnected to such devices to extend their 
55 capabilities, such as devices supporting an MPEG 7 description. MPEG-7 is the Moving Pictures Experts Group - 7, 
acting to standardize descriptions and description schemes for audiovisual information. The device may also be ex- 
tended to be personalized by multiple users, as desired. 

[0075] Because the description scheme is defined, the intelligent software agents can communicate among them- 
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selves to make intelligent inferences regarding the user's preferences. In addition, the development and upgrade of 
intelligent software agents for browsing and filtering applications can be simplified based on the standardized user 
description scheme. 

[0076] The description scheme is multi-modal in the following sense that it holds both high level (semantic) and low 
level features and/or descriptors. For example, the high and low level descriptors are actor name and motion model 
parameters, respectively. High level descriptors are easily readable by humans while low level descriptors are more 
easily read by machines and less understandable by humans. The program description scheme can be readily har- 
monized with existing EPG, PSIP, and DVB-SI information facilitating search and filtering of broadcast programs. Ex- 
isting services can be extended in the future by incorporating additional information using the compliant description 
scheme. 

[0077] For example, one case may include audiovisual programs that are prerecorded on a media such as a digital 
video disc where the digital video disc also contains a description scheme that has the same syntax and semantics of 
the description scheme that the FSB module uses. If the FSB module uses a different description scheme, atranscoder 
(converter) of the description scheme may be employed. The user may want to browse and view the content of the 
digital video disc. In this case, the user may not need to invoke the analysis module to author a program description. 
However, the user may want to invoke his or her user description scheme in filtering, searching and browsing the digital 
video disc content. Other sources of program information may likewise be used in the same manner. 
[0078] It is to be understood that any of the techniques described herein with relation to video are equally applicable 
to images (such as still image or a frame of a video) and audio (such as radio). 

[0079] An example of an audiovisual interface is shown in FIGS. 4-12 which is suitable for the preferred audiovisual 
description scheme. Referring to FIG. 4, by selecting the thumbnail function as a function of category provides a display 
with a set of categories on the left hand side. Selecting a particular category, such as news, provides a set of thumbnail 
views of different programs that are currently availableforviewing. In addition, the different programs may also include 
programs that will be available at a different time for viewing. The thumbnail views are short video segments that 
provide an indication of the content of the respective actual program that it corresponds with. Referring to FIG. 5, a 
thumbnail view of available programs in terms of channels may be displayed, if desired. Referring to FIG. 6, a text view 
of available programs in terms of channels may be displayed, if desired. Referring to FIG. 7, a frame view of particular 
programs may be displayed, if desired. A representative frame is displayed in the center of the display with a set of 
representative frames of different programs in the left hand column. The frequency of the number of frames may be 
selected, as desired. Also a set of frames are displayed on the lower portion of the display representative of different 
frames during the particular selected program. Referring to FIG. 8, a shot view of particular programs may be displayed, 
as desired. A representative frame of a shot is displayed in the center of the display with a set of representative frames 
of different programs in the left hand column. Also a set of shots are displayed on the lower portion of the display 
representative of different shots (segments of a program, typically sequential in nature) during the particular selected 
program. Referring to FIG. 9, a key frame view of particular programs may be displayed, as desired. A representative 
frame is displayed in the center of the display with a set of representative frames of different programs in the left hand 
column. Also a set of key frame views are displayed on the lower portion of the display representative of different key 
frame portions during the particular selected program. The number of key frames in each key frame view can be 
adjusted by selecting the level. Referring to FIG. 10, a highlight view may likewise be displayed, as desired. Referring 
to FIG. 11 , an event view may likewise be displayed, as desired. Referring to FIG. 12, a character/object view may 
likewise be displayed, as desired. 

[0080] An example of the description schemes is shown below in XML. The description scheme may be implemented 
in any language and include any of the included descriptions (or more), as desired. 

[0081] The proposed program description scheme includes three major sectionsfor describing a video program. The 
first section identifies the described program. The second section defines a number of views which may be useful in 
browsing applications. The third section defines a number of profiles which may be useful in filtering and search ap- 
plications. Therefore, the overall structure of the proposed description scheme is as follows: 
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<?XML version="1.0"> 

<!DOCTYPE MPEG- 7 SYSTEM "mpeg-7 . dtd"> 

<ProgramIdentity> 

<PrograraID> . . . </ProgramID> 
<PragramName> . . . </ProgramName> 
<SourceLocaticn> . . . </SourceLocation> 

</ProgramIdentity> 

<ProgramViews> 

<ThumbnailView> . . . </ThumbnailView> 
<SlideView> . . . </SlideView> 
<FrameView> . . . </ Frame View> 
<ShotView> . . . </ShotView> 
<KeyFrameView> . . . </KeyFrameView> 
<HighlightView> . . . </HighlightView> 
<EventView> . . . </EventView> 
<CloseCJpView> . . . </CloseOpView> 
<AlternataView> . . . </AlternateView> 

</ProgramViews> 

< ProgramProf iles > 

<GeneralProfile> . 
<CategoryProfile> 
<DateTimeProfile> 
<KeywordProfile> . 
<TriggerProfile> . . 
<StillProfile> ... 
<EventProf ile> . . . 
<CharacterProf ile> 
<ObjectProfile> . . 
<ColorProf ile> ... 
<TexcureProfile> . 
<ShapePro£iie> . . . 
<MotionProf ile> . . 



. </GeneralProfile> 
. . </CategoryProf ile> 
. . </DaCaTimeProfile> 
. </KeywordProfile> 
. </TriggerProfile> 
</StillProfile> 
</EventProfiXe> 
. . . </CharacterProfile> 
</ObjectProfile> 
</ColorProfile> 
. </TexcureProfile> 
</ShapeProfile> 
</MotionPro£ile> 



</ ProgramProf iles> 

Program Identity 
• Program ID 

<ProgramID> program-id </ProgramID> 



55 [0082] The descriptor <ProgramlD> contains a number or a string to identify a program. 
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• Program name 

<ProgramName> program-name </ProgramName> 
[0083] The descriptor <ProgramName> specifies the name of a program. 

• Source location 

<SourceLocation> source-url </SourceLocation> 
[0084] The descriptor <SourceLocation> specifies the location of a program in URL format. 

Program Views 

• Thumbnail view 

<Thu.iibnailView> 

<Image> thumbnail-image </Image> 
< /Thumbnai lView> 

[0085] The descriptor <ThumbnailView> specifies an image as the thumbnail representation of a program. 

• Slide view 

<SlideView> frame-id . . . </SlideView> 

[0086] The descriptor <SlideView> specifies a number of frames in a program which may be viewed as snapshots 
or in a slide show manner. 

• Frame view 

<FrameView> start-frame-id end-frame-id </FrameView> 



[0087] The descriptor <FrameView> specifies the start and end frames of a program. This is the most basic view of 
a program and any program has a frame view. 
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Shot view 



<ShotView> 

<Shot id»""> start-frame-id end-frame-id display-frame-id </Shot> 
<Shot id=""> start-frame-id end-frame-id display-frame-id </Shot> 



</ShotView> 



[0088] The descriptor <ShotView> specifies a number of shots in a program. The <Shot> descriptor defines the start 
'5 and end frames of a shot. It may also specify a frame to represent the shot. 



Key- frame view 



<KeyFrameView> 

<KeyFrames level=""> 

<Clip id=""> start- frame-id end-f rame-id display-frame-id </Clip> 
<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 



</KeyFrames> 
<KeyFrames level=""> 



<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 
<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 



</KeyFrameView> 

[0089] The descriptor <KeyFrameView> specifies key frames in a program. The key frames may be organized in a 
hierarchical manner and the hierarchy is captured by the descriptor <KeyFrames> with a level attribute. The clips which 
are associated with each key frame are defined by the descriptor <Clip>. Here the display frame in each clip is the 
corresponding key frame. 
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• Highlight view 

<HighlightView> 

<Highlight length=""> 

<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 
<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</Highlight> 

<Highlight length=""> 

<Clip id=""> start-frame-id end- frame-id display- frame- id </Clip> 
<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</Highlight> 

</HighlightView> 

[0090] The dssr-iptor <HighlightView> specifies clips to form highlights of a program. A program may have different 
versions of highlights which are tailored into various time length. The clips are grouped into each version of highlight 
which is specified by the descriptor <Highlight> with a length attribute. 

• Event view 



<Events name~""> 

<Clip id=""> start- frame-id end-frame-id display-frame-id </Clip> 
<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</Events> 
<Events name=""> 

<Clip id=""> start-frame-id end-frame-id display- frame- id </Clip> 
<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</Events> 

■ </EventView> 

[0091] The descriptor <EventView> specifies clips which are related to certain events in a program. The clips are 
grouped into the corresponding events which are specified by the descriptor <Event> with a name attribute. 
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Close-up view 



<CIose(JpView> 

<Target name=""> 

<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 
<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</Target> 
<Target name=""> 

<Clip id=""> start-frame-id end-frame-id display-frame- id </Clip> 
<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</Target> 

</CloseUpView> 

[0092] The descriptor <CloseUpView> specifies clips which may be zoomed in to certain targets in a program. The 
clips are grouped into the corresponding targets which are specified by the descriptor <Target> with a name attribute. 

• Alternate view 



<AlternateView> 

<AlternateSource id=""> source-url </AlternateSource> 
<AlternateSource id=""> source-url </AlternateSource> 

</AlternateView> 

[0093] The descriptor <AlternateView> specifies sources which may be shown as alternate views of a program. Each 
alternate view is specified by the descriptor <AlternateSource> with an id attribute. The locate of the source may be 
specified in URL format. 
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Program Profiles 

• General profile 

<GeneralProfile> 

<Title> title-text </Title> 
<Abstract> abstract-text </Abstract> 
<Audio> voice-annotation </Audio> 
<Www> web-page-url </Www> 
<ClosedCaption> yes/no </ClosedCaption> 
<Language> language-name </Language> 
<Rating> rating </Rating> 
<Length> time </Length> 
<Authors> author-name ... </Authors> 
<Producers> producer-name . . . </Producers> 
<Directors> director-name . . . </Directors> 
<Actors> actor-name . . . </Actors> 

</GeneralProfile> 

[0094] The descriptor <GeneralProfile> describes the general aspects of a program. 
• Category profile 
<CategoryProf ile> category-name . . . </CategoryProf ile> 

[0095] The descriptor <CategoryProfile> specifies the categories under which a program may be classified. 
« Date-time profile 

<OateTimeProfile> 

<ProductionDate> date </ProductionDate> 
<ReleaseDate> date </ReleaseDate> 
<RecordingDate> date </RecordingDate> 
<RecordingTime> time </RecordingTime> 

</DateTimePro£ile> 

[0096] The descriptor <DateTimeProfile> specifies various date and time information of a program. 
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• Keyword profile 

<KeywordProfxle> keyword . . . </ Keyword Prof ile> 

[0097] The descriptor <KeywordProfile> specifies a number of keywords which may be used to filter or search a 
program. 

• Trigger profile 

<TriggerProfile> trigger-f rame-id . . . </TriggerProf ile> 

[0098] The descriptor <TriggerProfile> specifies a number of frames in a program which may be used to trigger 
certain actions while the playback of the program. 

• Still profile 

<StillProfile> 

<Still id=""> 

<HotRegion id =""> 
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<Location> xl yl x2 y2 </Location> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 

<Wwv> web-page-url </Www> 
</HotRegion> 
<HotRegion id =""> 

<Location> xl yl x2 y2 </Location> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 

<Www> web-page-url </Www> 
</HotRegion> 

</Still> 
<Still id=""> 

<HotRegion id =""> 

<Location> xl yl x2 y2 </Location> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 

<Www> web-page-url </Www> 
</HotRegion> 
<HotRegion id =""> 

<Location> xl yl x2 y2 </Location> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 

<Www> web-page-url </www> 
</HotRegion> 

</Still> 

</StillProfiie> 

[0099] The descriptor <StillProf ile> specifies hot regions or regions of interest within a frame. The frame is specified 
by the descriptor <Still> with an id attribute which corresponds to the frame-id. Within a frame, each hot region is 
specified by the descriptor <HotRegion> with an id attribute. 



Event profile 



1 <EventProf ile> 

<EventList> event-name . . . </EventList> 
<Event name=""> 
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<Ww> web-page-url </Www> 
Occurrence ici=""> 

<Duration> start-frame-id end-frame-id </Duration> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
</Occurrence> 
<Occurrence id=""> 

<Duration> start-frame-id end- frame -id </Duration> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
</Occurrence> 

</Event> 
<£vent name=""> 

<Www> web-page-url </Www> 
<Occurrence id=""> 

<Duration> start-frame-id end-frame-id </Duration> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
</Occurrence> 
<Occurrence id=""> 

<Duration> start-frame-id end-frame-id </Duration> 

<Text> text-annotation </Text> 

<Audic> voice-annotation </Audio> 
</Occurrence> 



</EventProfile> 



[0100] The descriptor <EventProfile> specifies the detailed information for certain events in a program. Each event 
is specified by the descriptor <Event> with a name attribute. Each occurrence of an event is specified by the descriptor 
<Occurrence> with an id attribute which may be matched with a clip id under <EventView>. 



Character profile 



<CharacterProfile> 

<CharacterList> character-name ... </CharacterList> 
<Character name=""> 

<ActorName> actor-name </ActorName> 
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<Gender> male </Gender> 
<Age> age </Age> 
<Www> web-page-url </Www> 
<Occurrence id=""> 

<Duration> start-frame-id end-frame-id </Duration> 

<Location> frame: [xl yl x2 y2] ... </Location> 

<Motion> v„ v y v z v„ v 9 v v </Motion> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
</Occurre:'c;e.' 
<Occurrence id=""> 

<Duration> start-frame-id end-frame-id </Duration> 

<Location> frame: [xl yl x2 y2] ... </Location> 

<Motion> v„ v y v, v a v s v v </Motion> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
</Occurrence> 

</Character> 
<Character name=""> 

<ActorName> actor-name </ActorName> 

<Gender> male </Gender> 

<Age> age </Age> 

<Www> web-page-url </V?ww> 

<Occurrence id=""> 

<0uration> start-frame-id end-frame-id </Duration> 

<Location> frame: [xl yl x2 y2 ] ... </Location> 

<Motion> v„ v y v z v„ v, v y </Motion> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
</Occurrence> 
<Occurrence id-""> 

<Duration> start-frame-id end-frame- id </Duration> 

<Location> frame: [xl yl x2 y2] . . . ' </Location> 

<Motian> v, v y v t v 0 v, v Y </Motion> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
</Occurrence> 

</Character> 

</CharacterProfile> 

[0101] The descriptor <CharacterProfile> specif ies the detailed information for certain characters in a program. Each 
character is specified by the descriptor <Character> with a name attribute. Each occurrence of a character is specified 
by the descriptor <Occurrence> with an id attribute which may be matched with a clip id under <CloseUpView>. 
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Object profile 



<ObjectProfile> 

<ObjectList> object-naroe . . . </ObjectList> 
<Object name=""> 

<Www> web-page-url </Www> 
Occurrence id=""> 

<Duration> start-f rame-id end-frame-id </Duration> 

<Location> frame: [xl yl x2 y2] ... </Location> 

<Motion> v x v y v, v a v„ v Y </Motion> 

<Text> text -annotation </Text> 

<Audio> voice-annotation </Audio> 
</Occurrence> 
<Occurrence id=""> 

<Duration> start-f rarae-id e.nd-frame-id </Duration> 

<Location> frame: [xl yl x2 y2] ... </Location> 

<Motion> v, v y v z v„ v„ v v </Motion> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
</Occurrence> 

</Object> 
<Object name-""> 

<Www> web-page-url </Www> 
<Occurrence id=""> 

<Duration> start-frame-id end-frame-id </Duration> 

<Location> frame: [xl yl x2 y2] ... </Location> 

<Motion> v, v y v. v Q v, v v </Motion> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
< / Occur rence> 
<Gocurrence id-""> 

<Duration> start-frarae-id end-f rame-id </Duration> 

<Location> frame: [xl yl x2 y2) ... </Location> 

<Motion> v, v y v t v a v s v ¥ </Motion> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
</Occurrence> 



</Object> 
</ObjectProfile> 



[0102] The descriptor <ObjectProfile> specifies the detailed information forcertain objects in a program. Each object 
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is specified by the descriptor <Object> with a name attribute. Each occurrence of a object is specified by the descriptor 
<Occurrence> with an id attribute which may be matched with a clip id under <CloseUpView>. 

• Color profile 

<ColorProfile> 
</ColorProfile> 

[01 03] The descriptor <ColorProfile> specitiss ?Ke detailed color information of a program. All MPEG-7 color descrip- 
tors may be placed under here. 

• Texture profile 

<TextureProfile> 
</TextureProfile> 

[0104] The descriptor <TextureProfile> specifies the detailed texture information of a program. All MPEG-7 texture 
descriptors may be placed under here. 

• Shape profile 

<ShapeProfile> 
</ShapePro£ile> 

[0105] The descriptor <ShapeProfile> specifies the detailed shape information of a program. All MPEG-7 shape 
descriptors may be placed under here. 

• Motion profile 

<MotionProfile> 
</MotionProf ile> 

[0106] The descriptor <MotionProfile> specifies the detailed motion information of a program. All MPEG-7 motion 
descriptors may be. placed under here. 

User Description Scheme 

[0107] The proposed user description scheme includes three major sections for describing a user. The first section 
identifies the described user. The second section records a number of settings which may be preferred by the user. 
The third section records some statistics which may reflect certain usage patterns of the user. Therefore, the overall 
structure of the proposed description scheme is as follows: 
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<?XML version-"1.0"> 

<!DOCTVPE MPEG-7 SYSTEM "mpeg-7 . dtd"> 

<CJserIdentity> 

<UserID> . . . </UserID> 
<UserName> . . . </UserNair.e> 

</UserIdentity> 

<UserPreferences> 

<ErowsingPreferences> . . . </BrowsingPref en 
<FilteringPreferences> . . . </FilteringPref erences> 
<SearchPrelerences> . . . </3earcnPref erences> 
<DevicePreferences> . . . </DevicePref erences> 

</OserPreferences> 

<UserHistory> 

<BrowsingIlistcry> . . . </BrowsingHistory> 
<FilteringHistory> . . . </FilteringHistory> 
<SearchHistory> . . . </SearchHistory> 



<DeviceHistory> . . . </DeviceHistory> 
</UserHistory> 
<UserDemographics> 

<Age> . . . </Age> 

<Gender> ... </Gender> 

<2IP> . . . </ZIP> 
</OserDemographics> 



User Identity 



<UserID> user-id </UserID> 



[0108] The descriptor <UserlD> contains a number or a string to identify a user. 



<UserNanie> user-name </0serName> 



[01 09] The descriptor <UserName> specifies the name of a user. 
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User Preferences 

• Browsing preferences 

<BrowsingPreferencea> 
<Views> 

<ViewCategory id=""> view-id . . . </ViewCategory> 
<ViewCategory id=""> view- id . . . </ViewCategory> 

</Views> 

<FrameFrequency> frequency 
<ShotFrequency> frequency 
<KeyFrameLevel> level-id . 
<HighiightLength> length . 

</BrowsingPref erences> 



[0110] The descriptor <BrowsingPreferences> specifies the browsing preferences of a user. The user's preferred 
views are specified by the descriptor <Views>. For each category, the preferred views are specified by the descriptor 
<ViewCategory> with an id attribute which corresponds to the category id. The descriptor <FrameFrequency> specifies 
at what interval the frames should be displayed on a browsing slider under the frame view. The descriptor <ShotFre- 
quency> specifies at what interval the shots should be displayed on a browsing slider under the shot view. The descriptor 
<KeyFrameLevel> specifies at what level the key fiames should be displayed on a browsing slider under the key frame 
view. The descriptor <HighlightLength> specifies which version of the highlight should be shown under the highlight 
view. 

• Filtering preferences 

<FilteringPreferences> 

<Categories> category-name . . . </Categories> 
<Channels> channel -number . . . </Channels> 
<Ratings> rating-id . . . </Ratings> 
<Shows> show-name . . . </Shows> 
<Authors> author-name . . . </Authors> 
<Producers> producer-name . . . </Producers> 
<Directors> director-name . . . </Directors> 
<Rctors> actor-name . . . </Actors> 
<Keywords> keyword . . . </Keywords> 
<Titles> title-text ... </Titles? 

</FilteringPreferences> 
[0111] The descriptor <FilteringPreferences> specifies the filtering related preferences of a user. 



. . . <FrameFrequency> 
. . . <ShotFrequency> 
. . <KeyFrameLevel> 
. .<HighlightLength> 
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Search preferences 



<SearchPreferences> 



<:Categories> category-name . . . </Categories> 
<Channels> channel-number . . . </Channels> 
<Ratings> cating-id . . . </Ratings> 
<Shows> show-name ... </Shows> 



<Authors> author-name 
<Producers> producer-n 
<Directors> director-n 
<Actors> actor-name . . 
<Keyvords> keyword . . . 
<Titles> title-text . . 



. . . </Authors> 

ime . . . </Producers> 

ime . . . </Directors> 

</Actors> 
</Keyvords> 

</Titles> 



</SearchPreferences> 
[0112] The descriptor <SearchPreferences> specifies the search related preferences of a user. 



• Device preferences 

30 

<DevicePreferences> 

<Brightness> brightness-value </Brightness> 
35 <Contrast> contrast-value </Contrast> 

<Volume> volume-value </Volume> 
</DevicePreferences> 



■*o [0113] The descriptor <DevicePreferences> specifies the device preferences of a user. 



45 



55 
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Usage History 
Browsing history 



<BrowsingHistory> 
<Views> 

<ViewCategory id=""> view-id . . . </ViewCategory> 
<ViewCatego:y id=""> view-id ... </ViewCategory> 

</Views> 

<FrameFrequency> frequency . . . <FrameFrequency> 
<ShotFrequency> frequency . . . <ShotFrequency> 
<KeyFrameLevel> level-id . . . <KeyFrameLevel> 
<HighlightLength> length . . . <HighlightLength> 



< /Brows ingHistory> 

25 

[0114] The descriptor <BrowsingHistory> captures the history of a user's browsing related activities. 
30 • Filtering history 

<FilteringHistory> 

35 <Categories> category-name . . . </Categories> 

<Channels> channel-number . . . </Channels> 
<Ratings> rating-id . . . </Ratings> 
<Shows> show-name . . . </Shows> 
<Authors> author-name . . . </Authors> 

40 

<Producers> producer-name . . . </Producers> 
<Directors> director-name . . . </Directors> 
<Actors> actor-name . . . </Actors> 
<Keyword5> keyword . . . </Keywords> 
45 <Titles> title-text . . . </Titles> 

</FilteringHistory> 

so [0115] The descriptor <FilteringHistory> captures the history of a user's filtering related activities. 
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• Search history 

<SearchHistory> 

<Categories> category-name . . . </Categories> 
<Channels> channel-number . . . </Channels> 
<Ratings> rating-id . . . </Ratings> 
<Shows> show-name . . . </Showa> 
<Authors> author-name . . . </Authors> 
<Producers> producer-name . . . </Producers> 
<Directors> director-name ... </Directors> 

<Actors> actor-name </Actors> 

<Keywords> keyword . . . </Keywords> 
<Titles> title-text ... </Titles> 

</SearchHistory> 

[0116] The descriptor <SearchHistory> captures the history of a user's search related activities. 
• Device history 

< Devi ceHis to ry> 

<Brightness> brightness-value ... </Brightness> 
<Contrast> contrast-value . . . </Contrast> 
<volurae> volume-value ... </Volume> 

</DeviceHistory> 

[0117] The descriptor <DeviceHistory> captures the history of a user's device related activities. 

User demographics 
• Age 

<Age> age </Age> 
[01 18] The descriptor <Age> specifies the age of a user. 



<Gender> . . . </Gender> 



[0119] The descriptor <Gender> specifies the gender of a user. 
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• ZIP code 

<ZIP> . . . </ZIP> 

[0120] The descriptor <ZIP> specifies the ZIP code of where a user lives. 
System Description Scheme 

[01 21 ] The proposed system description scheme includes four major sections for describing a user. The first section 
identifies the described system. The second section keeps a list of ill known users. The third section keeps lists of 
available programs. The fourth section describes the capabilities of the system. Therefore, the overall structure of the 
proposed description scheme is as follows: 

<?XML version="1.0"> 

<!0OCTYPE MPEG-7 SYSTEM "mpeg-7 . dtd"> 
<SystemIdentity> 

<SystemID> . . . </SystenID> 

<SystemName> . . . </SystemName> 

<SystemSerialNumber> . . . </SystemSerialNumber> 
</SysteraIdentity> 
<SystemUsers> 

<Users> . . . </05ers> 
</SystemCJsers> 
<SystemPrograms> 

20 <Categories> ... </Categories> 

<Channels> . . . </Channels> 

<Programs> . . . </Programs> 
</SystemPrograms> 
<SystemCapabilities> 
25<Views> . .. </Views> 
</SystemCapabiXities> 

System Identity 

• System ID 

<SystemID> system-id </SystemID> 
[0122] The descriptor <SystemlD> contains a number or a string to identify a video system or device. 



• System name 



<SystemName> system-name </SystemName> 



EP1 158 795 A2 

[0123] The descriptor <SystemName> specifies the name of a video system or device. 
* System serial number 

<SystemSerialNumber> system-serial-number </SystemSerialNumber> 
[0124] The descriptor <SystemSerialNumber> specifies the serial number of a video system or device. 

System Users 

• Users 

<Users> . 

<Dser> 

<UserID> user-id </UserID> 

<UserName> user-name </UserName> 
</User> 
<User> 

<UserID> user-id </UserID> 
<UserName> user-name </0serName> 
</0ser> 

</U5ers> 

[0125] The descriptor <SystemUsers> lists a number of users who have registered on a video system or device. 
Each user is specified by the descriptor <User>. The descriptor <UserlD> specifies a number or a string which should 
match with the number or string specified in <UserlD> in one of the user description schemes. 



29 



EP1 158 795 A2 



Programs in the System 
• Categories 

<Categories> 

<Category> 

<CategoryID> category-id </CategorylD> 
<CategoryName> category-name </CategoryName> 
<SubCategories> sub-category-Ld . . . </SubCategories> 

</Category> 

<Category> 

<Ca tegoryID> category-id </CategoryID> 
<CategoryName> category-name </CategoryName> 
<SubCategories> sub-category-id ... </SubCategories> 

</Category> 

</Categories> 

[0126] The descriptor <Categories> lists a number of categories which have been registered on a video system or 
device. Each category is specified by the descriptor <Category>. The major-sub relationship between categories is 
captured by the descriptor < SubCategories>. 

• Channels 

<Channels> 

<Channel> 

<ChannelID> channel-id </ChannellO> 
<ChannelName> channel-name </ChannelName> 
<SubChannels> sub-channel -id . . . < /Subchannel s> 

</Channel> 

<Channel> 

<ChannelID> channel-id </ChannelID> 
<ChannelName> channel-name </ChannelName> 
<SubChannels> sub-channel-id . . . </SubChannels> 

</Channel> 

</Channels> 

[0127] The descriptor <Channels> lists a number of channels which have been registered on a video system or 
device. Each channel is specified by the descriptor <Channel>. The major-sub relationship between channels is cap- 
tured by the descriptor < SubChannels>. 
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Programs 



<Programs> 

<CategoryPrograms> 

<CategoryID> category-id </CategoryID> 
<Programs> program-id ... </Programs> 

</CategoryPrograms> 

<CategoryPrograms> 

<CategoryID> category-id </CategoryID> 
<Programs> program- id . .'. </Prograras> 

</CategoryPrograms> 

<CtiannelPrograms> 

<ChannelID> channel-id </ChannelID> 
<Programs> program- id . . . </Programs> . 

</ChannelPrograms> 

<ChannelPrograms> 

<ChannelID> channel-id </ChannelID> 
<Programs> program-id ... </Programs> 

</ChannelPrograms> 



</Programs> 

[0128] The descriptor <Programs> lists programs who are available on a video system or device. The programs are 
grouped under corresponding categories or channels. Each group of programs are specified by the descriptor <Cate- 
goryPrograms> or <ChannelPrograms>. Each program id contained in the descriptor <Programs> should match with 
the number or string specified in <ProgramlD> in one of the program description schemes. 



System Capabilities 



<Views> 

<View> 

<ViewID> view-id </ViewID> 
<ViewNarae> view-name </ViewName> 

</View> 

<View> 

<ViewID> view-id </ViewID> 
<ViewName> view-name </ViewNaxne> 
</View> 



[0129] The descriptor <Views> lists views which are supported by a video system or device. Each view is specified 
by the descriptor < View>. The descriptor <ViewName> contains a string which should match with one of the following 
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views used in the program description schemes: ThumbnailView, SlideView, FrameView, ShotView : KeyFrameView, 
HighlightView, EventView, and CloseUpView. 

[0130] The present inventors came to the realization that the program description scheme may be further modified 
to provide additional capabilities. Referring to FIG. 13, the modified program description scheme 400 includes four 
separate types of information, namely, a syntactic structure description scheme 402, a semantic structure description 
scheme 404, a visualization description scheme 406, and a meta information description scheme 408. It is to be un- 
derstood that in any particular system one or more of the description schemes may be included, as desired. 
[0131] Referring to FIG. 14, the visualization description scheme 406 enables fast and effective browsing of video 
program (and audio programs) by allowing access to the necessary data, preferably in a one-step process. The visu- 
alization description scheme 406 provides for several different presentations of the video content (or audio), such as 
for example, a thumbnail view description scheme 410, a key frame view description scheme 412, a highlight view 
description scheme 414, an event view description scheme 416, a close-up view description scheme 418, and an 
alternative view description scheme 420. Other presentation techniques and description schemes may be added, as 
desired. The thumbnail view description scheme 410 preferably includes an image 422 or reference to an image rep- 
resentative of the video content and a time reference 424 to the video. The key frame view description scheme 412 
preferably includes a level indicator 426 and a time reference 428. The level indicator 426 accommodates the pres- 
entation of a different number of key frames for the same video portion depending on the user's preference. The 
highlight view description scheme 414 includes a length indicator 430 and a time reference 432. The length indicator 
430 accommodates the presentation of a different highlight duration of a video depending on the user's preference. 
The event view description scheme 41 6 preferably includes an event indicator 434 for the selection of the desired event 
and a time reference 436. The close-up view description scheme 41 8 preferably includes a target indicator 438 and a 
time reference 440. The alternate view description scheme preferably includes a source indicator 442. To increase 
performance of the system it is preferred to specify the data which is needed to render such views in a centralized and 
straightforward manner. By doing so, it is then feasible to access the data in a simple one-step process without complex 
parsing of the video. 

[0132] Referring to FIG. 15, the meta information description scheme 408 generally includes various descriptors 
which carry general information about a video (or audio) program such as the title, category, keywords, etc. Additional 
descriptors, such as those previously described, may be included, as desired. 

[0133] Referring again to FIG. 13, the syntactic structure description scheme 402 specifies the physical structure of 
a video program (or audio), e.g., a table of contents. The physical features, may include for example, color, texture, 
motion, etc. The syntactic structure description scheme 402 preferably includes three modules, namely a segment 
description scheme 450, a region description scheme 452, and a segment/region relation graph description scheme 
454. The segment description scheme 450 may be used to define relationships between different portions of the video 
consisting of multiple frames of the video. A segment description scheme 450 may contain another segment description 
scheme 450 and/or shot description scheme to form a segment tree. Such a segment tree may be used to define a 
temporal structure of a video program. Multiple segment trees may be created and thereby create multiple table of 
contents. For example, a video program may be segmented into story units, scenes, and shots, from which the segment 
description scheme 450 may contain such information as a table of contents. The shot description scheme may contain 
a number of key frame description schemes, a mosaic description scheme(s), a camera motion description scheme 
(s), etc. The key frame description scheme may contain a still image description scheme which may in turn contains 
color and texture descriptors. It is noted that various low level descriptors may be included in the still image description 
scheme under the segment description scheme. Also, the visual descriptors may be included in the region description 
scheme which is not necessarily under a still image description scheme. On example of a segment description scheme 
450 is shown in FIG. 16. 

[0134] Referring to FIG. 17, the region description scheme 452 defines the interrelationships between groups of 
pixels of the same and/or different frames of the video . The region description scheme 452 may also contain geometrical 
features, color, texture features, motion features, etc. 

[0135] Referring to FIG. 1 8, the segment/region relation graph description scheme 454 defines the interrelationships 
between a plurality of regions (or region description schemes), a plurality of segments (or segment description 
schemes), and/or a plurality of regions (or description schemes) and segments (or description schemes). 
[01 36] Referring again to FIG. 1 3, the semantic structure description scheme 404 is used to specify semantic features 
of a video program (or audio), e.g. semantic events. In a similar manner to the syntactic structure description scheme, 
the semantic structure description scheme 404 preferably includes three modules, namely an event description scheme 
480, an object description scheme 482, and an event/objection relation graph description scheme 484. The event 
description scheme 480 may be used to form relationships between different events of the video normally consisting 
of multiple frames of the video. An event description scheme 480 may contain another event description scheme 480 
to form a segment tree. Such an event segment tree may be used to define a semantic index table for a video program. 
Multiple event trees may be created and thereby creating multiple index tables. For example, a video program may 
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include multiple events; such as a basketball ciunK, a fast break, and a free throw, and the event description scheme 
may contain such information as an index table. The event description scheme may also contain references which link 
the event to the corresponding segments and/or regions specified in the syntactic structure description scheme. On 
example of an event description scheme is shown in FIG. 1 9. 

[0137] Referring to FIG. 20, the object description scheme 482 defines the interrelationships between groups of 
pixels of the same and/or different frames of the video representative of objects. The object description scheme 482 
may contain another object description scheme, and thereby form an object tree. Such an object tree may be used to 
define an object index table for a video program. The object description scheme may also contain references which 
link the object to the corresponding segments and/or regions specified in the syntactic structure description scheme. 
[0138] Referring to FIG. 21 , the event/object relation graph description scheme 484 defines the interrelationships 
between a plurality of events (or event description schemes), a plurality of objects (or object description schemes), 
and/or a plurality of events (or description schemes) and objects (or description schemes). 

[0139] After further consideration, the present inventors came the realization that the particular design of the user 
preference description scheme is important to implement portability, while permitting adaptive updating, of the user 
preference description scheme. Moreover, the user preference description scheme should be readily usable by the 
system while likewise being suitable for modification based on the user's historical usage patterns. It is possible to 
collectively track all users of a particular device to build a database for the historical viewing preferences of the users 
of the device, and thereafter process the data dynamically to determine which content the users would likely desire. 
However, this implementation would require the storage of a large amount of data and the associated dynamic process- 
ing requirements to determine the user preferences. It is to be understood that the user preference description scheme 
mriy be used alone or in combination with other description scheme. 

[0140] Referring to FIG. 22, to achieve portability and potentially decreased processing requirements the user pref- 
erence description scheme 20 should be divided into at least two separate description schemes, namely, a usage 
preference description scheme 500 and a usage history description scheme 502. The usage preference description 
scheme 500. described in detail later, includes a description scheme of the user's audio and/or video consumption 
preferences. The usage preference description scheme 500 describes one or more of the following, depending on the 
particular implementation, (a) browsing preferences, (b) filtering preferences, (c) searching preferences, and (d) device 
preferences of the user. The type of preferences shown in the usage preference description scheme 500 are generally 
immediately usable by the system for selecting and otherwise using the available audio and/or video content. In other 
words, the usage preference description scheme 500 includes data describing audio and/or video consumption of the 
user. The usage history description scheme 502, described in detail later, includes a description scheme of the user's 
historical audio and/or video activity, such as browsing, device settings, viewing, and selection. The usage history 
description scheme 502 describes one or more of the following, depending on the particular implementation, (a) brows- 
ing history, (b) filtering history, (c) searching history, (d) device usage history, and (e) the time of action of the imple- 
mentation. The type of preferences shown in the usage history description scheme 502 are not generally immediately 
usable by the system for selecting and otherwise using the available audio and/or video content. The data contained 
in the usage history description scheme 502 may be considered generally "unprocessed", at least in comparison to 
the data contained in the usage preferences description scheme 500 because it generally contains the historical usage 
data of the audio and/or video content of the viewer. 

[0141] In general, storing the user's usage history including facts that the user viewed and selected programs and 
browsing procedures thereof viewed, and utilizing a variety of algorithms, a machine may automatically prepare the 
user's preferences. Utlizing the user's history description scheme may update the user's preference description. As 
an example, taking statistics of such history information, user's preference information may be derived. 
[0142] Since history information and preference information are independently managed, the preference information 
may be updated as desired. Thus, a machine less capable of preparing user's preference information —e.g., a mobile 
terminal— may only store the history information, then transmitthe history information to another machine morecapable 
of preparing preference information so as to update the preference information. Further, a user's taste information may 
be derived from a plurality of history informations of the same user. 

[0143] Furthermore, since the machine contains the user's viewing history informations and user 1 s preference in- 
formations, when restarting the machine, the following program or content may be automatically provided to the user 
for viewing, and a new program may be recommended based on the preference information. 

[0144] After consideration of the usage preference description 500 and the usage history description 502, the present 
inventors came to the realization that in the home environment many different users with different viewing and usage 
preferences may use the same device. For example, with a male adult preferring sports, a female adult preferring 
afternoon talk shows, and a three year old child preferring children's programming, the total information contained in 
the usage preference description 500 and the usage history description 502 will not be individually suitable for any 
particular user. The resulting composite data and its usage by the device is frustrating to the users because the device 
will not properly select and present audio and/or video content that is tailored to any particular user. To alleviate this 
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limitation, the user preference description 20 may also include a user identification (user identifier) description 504. 
The user identification description 504 includes an identification of the particular user that is using the device. By 
incorporating a user identification description 504 more than one user may use the device while maintaining a different 
or a unique set of data within the usage preference description 500 and the usage history description 502. Accordingly, 

5 the user identification description 504 associates the appropriate usage preference description(s) 500 and usage his- 
tory description(s) 502 for the particular user identified by the user identification description 504. With multiple user 
identification descriptions 504, multiple entries within a single user identification description 504 identifying different 
users, and/or including the user identification description within the usage preference description 500 and/or usage 
history description 502 to provide the association therebetween, multiple users can readily use the same device while 

10 maintaining their individuality. Also, without the user identification description in the preferences and/or history, the 
user may more readily customize content anonymously. In addition, the user's user identification description 504 may 
be used to identify multiple different sets of usage preference descriptions 500 -- usage history descriptions 502, from 
which the user may select for present interaction with the device depending on usage conditions. The use of multiple 
user identification descriptions for the same user is useful when the user uses dultiple different types of devices, such 

15 as a television, a home stereo, a business television, a hotel television, and a vehicle audio player, and maintains 
multiple different sets of preference descriptions. Further, the identification may likewise be used to identify groups of 
individuals, such as for example, a family. In addition, devices that are used on a temporary basis, such as those in 
hotel rooms or rental cars, the user identification requirements may be overridden by employing a temporary session 
user identification assigned by such devices. In applications where privacy concerns may be resolved or are otherwise 

20 not a concern, the user identification description 504 may also contain demographic information of the user. In this 
manner, as the usage history description 502 increases during use overtime, this demographic data and/or data re- 
garding usage patterns may be made available to other sources. The data may be used for any purpose, such as for 
example, providing targeted advertising or programming on the device based on such data. 

[01 45] Referring to FIG. 23, periodically an agent 51 0 processes the usage history description(s) 502 for a particular 

25 user to "automatically" determine the particular user's preferences. In this manner, the user's usage preference de- 
scription 500 is updated to reflect data stored in the usage history description 502. This processing by the agent 510 
is preferably performed on a periodic basis so that during normal operation the usage history description 502 does not 
need to be processed, or otherwise queried, to determine the user's current browsing, filtering, searching, and device 
preferences. The usage preference description 500 is relatively compact and suitable for storage on a portable storage 

30 device, such as a smart card, for use by other devices as previously described. 

[0146] Frequently, the user may be traveling away from home with his smart card containing his usage preference 
description 500. During such traveling the user will likely be browsing, filtering, searching, and setting device prefer- 
ences of audio and/or video content on devices into which he provided his usage preference description 500. However, 
in some circumstances the audio and/or video content browsed, filtered, searched, and device preferences of the user 

35 may not be typically what he is normally interested in. In addition, for a single device the user may desire more than 
one profile depending on the season, such as football season, basketball season, baseball season, fall, winter, summer, 
and spring. Accordingly, it may not be appropriate forthe device to create a usage history description 502 and thereafter 
have the agent 51 0 "automatically" update the user's usage preference description 500. This will in effect corrupt the 
user's usage preference description 500. Accordingly, the device should include an option that disables the agent 51 0 

40 from updating the usage preference description 500. Alternatively, the usage preference description 500 may include 
one or more fields or data structures that indicate whether or not the user desires the usage preference description 
500 (or portions thereof) to be updated. 

[01 47] Referring to FIG. 24, the device may use the program descriptions provided by any suitable source describing 
the current and/or future audio and/or video content available from which a filtering agent 520 selects the appropriate 
■*5 content for the particular user(s). The content is selected based upon the usage preference description for a particular 
user identification(s) to determine a list of preferred audio and/or video programs. 

[01 48] As it may be observed, with a relatively compact user preference description 500 the user's preferences are 
readily movable to different devices, such as a personal video recorder, a TiVO player, a Replay Networks player, a 
car audio player, or other audio and/or video appliance. Yet, the user preference description 500 may be updated in 

so accordance with the user's browsing, filtering, searching, and device preferences. 

[0149] Referring to FIG. 25, the usage preference description 500 preferably includes three different categories of 
descriptions, depending on the particular implementation. The preferred descriptions include (a) browsing preferences 
description 530, (b) filtering and search preferences description, 532 and (c) device preferences description 534. The 
browsing preferences description 530 relates to the viewing preferences of audio and/or video programs. The filtering 

55 and search preferences description 532 relates to audio and/or video program level preferences. The program level 
preferences are not necessarily used at the same time as the (browsing) viewing preferences. For example, preferred 
programs can be determined as a result of filtering program descriptions according to user's filtering preferences. A 
particular preferred program may subsequently be viewed in accordance with user's browsing preferences. Accordingly, 
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efficient implementation may be achieved if the browsing preferences description 530 is separate, at least logically, 
from the filtering and search preferences description 532. The device preferences description 534 relates to the pref- 
erences for setting up the device in relation to the type of content being presented, e.g. romance, drama, action, 
violence, evening, morning, day, weekend, weekday, and/or the available presentation devices. For example, presen- 
s tation devices may include stereo sound, mono sound, surround sound, multiple potential displays, multiple different 
sets of audio speakers, AC-3, and Dolby Digital. It may likewise be observed that the device preferences description 
534 is likewise separate, at least logically, from the browsing description 530 and filtering/search preferences descrip- 
tion 532. 

[0150] The browsing preferences description 530 contains descriptors that describe preferences of the user for 
10 browsing multimedia (audio and/or video) information. In the case of video, for example, the browsing preferences may 
include user's preference for continuous playback of the entire program versus visualizing a short summary of the 
program. Various summary types may be described in the program descriptions describing multiple different views of 
programs where these descriptions are utilized by the device to facilitate rapid non-linear browsing, viewing, and nav- 
igation. Parameters of the various summary types should also be specified, i.e., number of hierarchy levels when the 
is keyframe summary is preferred, or the time duration of the video highlight when highlight summary is preferred. In 
addition, browsing preferences may also include descriptors describing parental control settings. A switch descriptor 
(set by the user) should also be included to specify whether or not the preferences can be modified without consulting 
the userfirst. This prevents inadvertent changing or updating of the preferences by the device. In addition, it is desirable 
that the browsing preferences are media content dependent. For example, a user may prefer 1 5 minute video highlight 
20 of a basketball game or may prefer to see only the 3-point shots. The same user may prefer a keyframe summary with 
two levels of hierarchy for home videos. 

[0151] The filtering and search preferences description 532 preferably has four descriptions defined therein, depend- 
ing on the particular embodiment. The keyword preferences description 540 is used to specify favorite topics that may 
not be captured in the title, category, etc., information. This permits the acceptance of a query for matching entries in 

25 any of the available data fields. The content preferences description 542 is used to facilitate capturing, for instance, 
favorite actors, directors. The creation preferences description 544 is used to specify capturing, for instance, titles of 
favorite shows. The classification preferences description 546 is used to specify descriptions, for instance, a favorite 
program category. A switch descriptor, activated by the user, may be included to specify whether or not the preferences 
may be modified without consulting the user, as previously described. 

30 [0152] The device preferences description 534 contains descriptors describing preferred audio and/or video render- 
ing settings, such as volume, balance, bass, treble, brightness, contrast, closed captioning, AC-3, Dolby digital, which 
display device of several, type of display device, etc. The settings of the device relate to how the user browses and 
consumes the audio and/or video content. It is desirable to be able to specify the device setting preferences in a media 
type and content-dependent manner. For example the preferred volume settings for an action movie may be higher 

35 than a drama, or the preferred settings of bass for classical music and rock music may be different. A switch descriptor, 
activated by the user, may be included to specify whether or not the preferences may be modified without consulting 
the user, as previously described. 

[0153] Referring to FIG. 26, the usage preferences description maybe used in cooperation with an MPEG-7 compliant 
data stream and/or device. MPEG-7 descriptions are described in ISO/I EC JTC1/SC29/WG11 "MPEG-7 Media/Meta 

40 DSs (V0.2), August 1999, incorporated by reference herein. It is preferable that media content descriptions are con- 
sistent with descriptions of preferences of users consuming the media. Consistency can be achieved by using common 
descriptors in media and user preference descriptions or by specifying a correspondence between user preferences 
and media descriptors. Browsing preferences descriptions are preferably consistent with media descriptions describing 
different views and summaries of the media. The content preferences description 542 is preferably consistent with, e. 

45 g.,a subset of the content description of the media 552 specified in MPEG-7 by content description scheme. The 
classification preferences description 544 is preferably consistent with, e.g., a subset of the classification description 
554 defined in MPEG-7 as classification description scheme. The creation preferences description 546 is preferably 
consistent with, e.g., a subset of the creation description 556 specified in MPEG-7 by creation description scheme. 
The keyword preferences description 540 is preferably a string supporting multiple languages and consistent with 

so corresponding media content description schemes. Consistency between media and user preference descriptions is 
depicted or shown in FIG. 26 by couble arrows in the case of content, creation, and classification preferences. 
[0154] Referring to FIG. 27, the usage history description 502 preferably includes three different categories of de- 
scriptions, depending on the particular implementation. The preferred descriptions include (a) browsing history de- 
scription 560, (b) filtering and search history description 562, and (c) device usage history description 564, as previously 

55 described in relation to the usage preference description 500. The filtering and search history description 562 preferably 
has four descriptions defined therein, depending on the particular embodiment, namely, a keyword usage history de- 
scription 566, a content usage history description 568, a creation preferences description 570, and a classification 
usage history description 572, as previously described with respect to the preferences. The usage history description 
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502 may contain additional descriptors therein (oi description if desired) that describe the time and/or time duration of 
information contained therein. The time refers to the duration of consuming a particular audio and/or video program. 
The duration of time that a particular program has been viewed provides information that may be used to determine 
user preferences. For example, if a user only watches a show for 5 minutes then it may not be a suitable preference 
for inclusion the usage preference description 500. In addition, the present inventors came to the realization that an 
even more accurate measure of the user's preference of a particular audio and/or video program is the time viewed in 
light of the total duration of the program. This accounts for the relative viewing duration of a program. For example 
watching 30 minutes of a 4 hour show may be of less relevance than watching 30 minutes of a 30 minute show to 
determine preference data for inclusion in the usage preference description 500. 

[0155] Referring to FIG. 28, an exemplary example of an audio and/or video program receiver with persistent storage 
601 is illustrated. As shown, audio/video program descriptions 600 are available from the broadcast or other source, 
such as a telephone line. The user preference 606 description facilitate personalization of the browsing 603, filtering 
and search 604, and device settings in the display 602. In this embodiment, the user preferences 606 are stored at 
the user's terminal with provision for transporting it to other systems, for example via a smart card 607. Alternatively, 
the user preferences 606 may be stored in a server and the content adaptation can be performed according to user 
descriptions at the server and then the preferred content is transmitted to the user. The user 61 1 may directly provide 
the user preferences 606, if desired. The user preferences 606 and/or user history 609 may likewise be provided to a 
service provider 61 0. The system may employ an application that records user's usage history 609 in the form of usage 
hislory description, as previously defined. The usage history description is then utilized by another application, e.g., a 
smart agent, to automatically (608) map usage history 609 to user preferences 606. 

Additional Attributes and Descriptors 

In The Description and The Description Scheme 

[0156] The present inventors came to the realization that additional functionality for the system may be achieved by 
the incorporation of particular types of information in the descriptions and description schemes. A description scheme 
is a data model of descriptions. It specif ies the descriptors and their syntax as they are used in the description. In what 
follows, use the terms description and description scheme may be used interchangeably since they both correspond 
to describing media and user preferences. An explanation of the additional attributes and descriptors in the descriptions 
will be provided, followed by an example of portions of example descriptions. 

[0157] After further consideration, there is a need for many users to maintain multiple separate user preference 
descriptions. Multiple user preference descriptions may correspond to, for example, different locations (e.g., at home, 
at the office, away from home, stationary versus traveling in a vehicle), different situations, differenttimes (e.g., different 
days,differentseasons), different emotional states of the user (e.g., happy mood versus tired or sad), and/or persistence 
(e.g., temporary usage versus permanent usage). Further, the user preference descriptions may include differentiation 
for different terminals with different primary functionalities (e.g., a personal video recorder versus a cell phone). In 
addition, available communication channel bandwidth at different locations or situations may use different preferences. 
Also, the preference of a user for the length of an audiovisual summary of a video program for downloading may be 
different. The user in different usage conditions may use the user identification description scheme as a basis to dis- 
tinguish between different devices and/or services. An example of different conditions may include a television broad- 
cast receiver and a cellular telephone. 

[0158] In addition to maintaining multiple user preferences for a particular user based on the aforementioned con- 
ditions, the present inventors also came to the realization that the different locations, different situations, different 
emotional states, different seasons, and/or different terminals (etc.), may likewise be used as the basis fordistinguishing 
between the user preference descriptions. 

[0159] One technique to permit a particular user to have multiple preference descriptions and distinguishing them 
from one another is by using different usernames or by using a versioning mechanism, such as a version descriptor 
in the identification description scheme, as described later. 

[0160] As previously described, the system may include multiple user preference descriptions for a particular user. 
With multiple descriptions, the system may express the different user preferences with different granularity, e.g., a 
greater or lesser amount of detail. The increased granularity (sparseness) may be merely the result of applying a filter 
to the user preference description that further reduces the amount of data. In other words, the structure of the usage 
preference description may be identical with the difference being the result of the filter further reducing the data. In 
another embodiment, the variable granularity results in a different size of the data contained in the user preferences, 
which may be based upon, if desired, the location and/or application of the user. User preferences with increased 
granularity may be especially suitable for storage on portable memory devices with limited memory capability. Likewise, 
the granularity may be applied to the usage history. 
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[0161] Another aspect of the present invention permits the user preferences (and history) to be based upon the 
media type, media source, or content (e.g., music versus video, radio versus television broadcast, and/or sports video 
versus home video). These preferences relate to the audio and/or video itself, as opposed to a third party characteri- 
zation of the desirability of the multimedia. The inclusion of this information permits a reduction in the computational 

s processing requirements depending on the media type, media source, and/or content of the media. 

[0162] Another feature that may be included in the system is a protection attribute for each, or a selected set of, 
component of the user descriptions. The protection attributes specifies the access right of a system or service provider, 
typically a party other than the user himself, to the user's descriptions or any component thereof. In one embodiment, 
the protection attributes may specify the user's desire to permit others access to such data. One technique to implement 

10 the protection attribute is to include a protection attribute as a primitive attribute that is contained by all relevant parts 
of the user description scheme. 

[0163] Descriptors and description schemes for browsing preferences may be aligned with particular types of mul- 
timedia summary description schemes that are contained in ISO/I EC JTC1/SC29/WG1 1 N3246, "MPEG-7 Generic AV 
Description Schemes, Working Draft v2.0", Noordwijkerhout, March 2000. This allows the user to specify the type of 

15 a particular visual summary of an audiovisual program, and the duration of a summary that is in the form of a visual 
highlight. However, after further consideration the present inventors have determined that specification of the preferred 
minimum and maximum amount of data permitted in an audiovisual summary significantly enhances the system ca- 
pability. Such a provision provides, for example, the capability of the user effectively browsing audiovisual summaries 
of content over channels with limited bandwidth and using terminals with different limitations. With a terminal connected 

20 to a bandwidth limited channel, the user may specify preference for a relatively short highlight of the program, while 
with a terminal that is connected to a higher bandwidth channel, the user may specify preference for a longer highlight 
of the program. Such a set of channels may be mobile channels and cable channels. In addition, for terminals that are 
not capable of displaying frames at a video rate, the user may prefer keyframe summaries consisting of a maximum 
number of keyframes appropriate for the communication channel bandwidth. To achieve these enhancements, the 

25 present inventors propose using descriptors in the browsing preferences description (and description scheme, or other 
preferences description) specifying the minimum, maximum; and exact number of keyframes, and minimum, maximum, 
and exact duration of audio and/or visual highlights. 

[0164] As described, the description scheme is adaptable to express the preferred minimum and maximum amount 
of visual material to adapt to different viewing preferences as well as terminal and communication channel bandwidth 

JO limitations. This implementation may be achieved by the following descriptors included in the browsing preferences 
description scheme: MaxN urn Of Keyframes, MinNumOfKeyframes, NumOf Keyframes, MaxSummaryDuration, Min- 
SummaryDuration, and SummaryDuration. The MaxNumOfKeyframes and MinNumofKeyframes preference descrip- 
tors specify, respectively, the maximum and minimum number of keyframes in the keyframe-summary of a video pro- 
gram. NumOfKeyf rames descriptor specifies the standard number of keyframes. Depending on the known bandwidth 

35 conditions of a known connection that the user uses regularly, he or she may specify these descriptors. The MaxSum- 
maryDuration and MinSummaryDuration descriptors specify, respectively, the maximum and minimum temporal dura- 
tion of an audiovisual highlight summary. SummaryDuration descriptor specifies the standard duration of highlight- 
summary. Again, depending on user's taste terminal, and channel limitations, the user may specify these descriptor. 
The MaxSummaryDuration and MinSummaryDuration descriptors apply to preferences for audio signals as well as 

■w where audio highlights may have been generated by audio skimming methods. User's browsing preference descriptions 
may be correlated with media descriptions by a filtering agent 520 in Fig. 24 in order to determine media descriptions 
that contain summary descriptions that match user's preference descriptions and provide the user the associated sum- 
marized media in the preferred type of summary. 

[0165] An additional descriptor that may be introduced is an abstraction fidelity descriptor for universal multimedia 
■** access application, where fidelity of a summary abstraction of a program is described. This can correspond to the 
variation fidelity descriptor defined in ISO/I EC JTC1/SC29 WG11 N3246, "MPEG-7 Multimedia Description Schemes, 
Working Draft v2.0", Noordwijkerhout, March 2000. This provides an alternative to the explicit specification of the du- 
ration and bounds on the number of keyframes. A Segment Theme descriptor(s) may describe the preferred theme, 
or point of view, of a segment, e.g., a video or audio clip, annotated with its theme or emphasis point. For example, 
so the theme may specify characteristics of the content of the theme. Such characterization may include a goal from your 
favorite team, 3-point shots from yourfavorite player, etc. Specifying these descriptor(s) and also ranking them enables 
a client application or a server to provide to the user segments according to preferred themes (and/or their ranking) 
matching to the their labels or descriptors at the segment level, or provide users with pre-assembled highlights com- 
posed of segments with labels matching the SegmentTheme preference. 
55 [0166] Existing filtering and search user preference descriptions are directed to techniques of using the audiovisual 
content in an effective manner by finding, selecting and consuming the desired audiovisual material, while focusing on 
the content of the audiovisual materials. While such descriptions are beneficial, the present inventors came to the 
further realization that the identification of the source of the material , in contrast to merely its content, provides beneficial 
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information for the processing and presentation of the audiovisual materials. For example, the source of the content 
may be from terrestrial sources, digital video disc, cable television, analog broadcast television, digital broadcast tel- 
evision, analog radio broadcasts, and digital radio broadcasts. The inclusion of this information permits the user to 
select among these different sources and increase effectiveness by narrowing down the choices to those sources that 
s are available to the user, such as terrestrial broadcast which is more widely available than satellite broadcast. For 
example, user may describe user's preference for "Star Trek" episodes that are available from terrestrial broadcast 
channels only. 

[0167] This source distinction and identification may be performed by including a source preferences description 
scheme under the filtering and search preferences description scheme (or other description scheme). Accordingly, the 
10 search and preferences description scheme may include from zero or one (or more if desired) source preferences 
description scheme. The source preferences description scheme may be derived from the Media Format description 
scheme or Publication Description Scheme specified in ISO/IEC JTC1/SC29/WG11 N3247, MPEG-7 Multimedia De- 
scription Schemes, Experimentation Model (v2.0) Noordwijkerhout, March 2000. 

[01 68] Anotherfeature that may be included in the system, in addition to the user's preferences, is the user's negative 
'5 preferences. The negative preferences may include the user's dislikes and their relative rankings. By specifying the 
negative preferences, the system is less likely to select such matching preferences. This may be implemented, for 
example, by permitting positive and negative values to the preferencevalue descriptor. 

[01 69] Another feature that may be included in the system is the specification of the user's preferences as a relative 
preference measure of a particular set of user preferences with respect to another set of preferences, such as for 
20 example, by using BetterThan and WorseThan descriptors. This permits an implicit relative ranking of preferences 
even in the absence of a preference value descriptor for each preference set. This may be implemented, for example, 
by including Betterthan and WorseThan descriptors in the filtering and search preferences descriptions. 

Expression of the Additional Attributes 

25 

[0170] The following descriptions are expressed in XML (Extensible Markup Language), incorporated by reference 
herein. It is to be understood that any other description language may likewise be used. 
[0171] The definition of the user preference description may be as follows. 

30 

<UserPreference> 

<UserIdentificr protection-'true" userName="paul"/> 
<UsagePreferences alIowAutomaticUpdate="false"> 

35 

<BrowsingPreferences> 



</BrowsingPreferences> 
<FilteringAndSearchPreferences> 



</FilteringAndSearchPreferences> 
<DevicePreferences> 



</DevicePreferences> 
</UsageHistory> 

50 

</UsageHistory> 
</UserPreference> 

55 

[0172] The primitive attributes "protection" and "allowAutomaticUpdate" may be instantiated in the Userldentifier, 
Usage Preferences, and Usage History descriptions and all its relevant parts, namely, in Browsing Preferences de- 
scription, Filtering and Search Preferences description, Device Preferences description, and sub-description schemes 
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of the Usage History description Scheme. 

[0173] The "allowAutomaticllpdate" attribute (set by the user) should be included in a description scheme specifying 
whether or not the preferences can be automatically modified (e.g. , by an agent utilizing the usage history description) 
without consulting with the user. 

5 [0174] The protection attribute should be included in a description specifying whether the user allows the system to 
make preference/history public or not. When the user agrees to make some parts of his preference/history public, for 
example, to service providers, the service providers can collect this information and then serve to the user contents 
that are tailored to the user's history/preferences. In the above example description, the user prefers to keep his user- 
name private. He also does not wish the system to automatically update his preferences. 

10 [0175] The user identification description serves the purpose of an identifier that distinguishes a particular instanti- 
ation of the user description scheme from other instantiations for other users or other instantiations for the same user 
for different usage conditions and situations: 

[0176] Tho username descriptor may identify a specific user from other users. In a home setting, each member of 
the household may be identified using a username that is unique in the household for all devices that the members of 
15 that household use on a regular basis. A username can also be used to distinguish the user description scheme of not 
only an individual but also a group of people, e.g., the family. Those devices that are used on a temporary basis, 
potentially by many different people, (such as those in hotel rooms or rental cars) may assign temporary session 
identifications to ensure uniqueness of identifications. 

[0177] Alternatively, a version descriptor may also be included in the user identifier description to define different 
20 versions of the user descriptions (preferences and usage history) associated with a particular username. Through the 
mechanism of the version, a person can specify different preferences and usage history, corresponding to different 
locations (at home, at the office, away from home, stationary versus traveling in a vehicle), different situations, different 
emotional states (happy versus sad), different seasons, etc. Different user descriptions are distinguished by distinct 
version descriptors. The type of the version descriptor, may be for example, an integer, a string, or expressed as an 
25 attribute of the user identification description scheme. 

[0178] The usage preference description may include a PreferenceType description, distinguishing a particular set 
of preferences or history according to time, or place, or a place and time combination. The definition of the usage 
preference description may be as shown in the following example, where place is "office" and time period is "8 hours 
starting from 8 AM" 

30 



<PreferenceType> 
<Place> 

<PlaceName xmi:lang="en">Office</PlaceName> 
</Placc> 
<Time> 
<TimePoint> 
<h>8</h> 
</TimePoint> 
<Duration> 
<No_h>8</No_h> 
</Duration> 
</Time> 
</PrefeienceType> 



[0179] The preferencetype descriptor may be used to identify the preference type of one or more set of preferences. 
As previously described, a user may have different preferences depending on the user's situation, location, time, sea- 
son, and so on. 

[0180] The browsing preferences description may describe preferences of the user for browsing multimedia infor- 
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mation. In essence, this description expresses the user's preferences for consuming (viewing, listening) a multimedia 
information. This browsing preferences description may include for example, a Summary Preferences description. The 
browsing preferences description may include in the case of video, for example, the user's preferences for continuous 
playback of the entire program versus visualizing a short summary of the program. Various summary types are specified 
in the Summary Description Scheme in ISO/I EC JTC1/SC29 WG1 1 N3246, "MPEG-7 Multimedia Description Schemes, 
Working Draft v2.0", Noordwijkerhout, March 2000, including a keyframe summary, a highlight summary, etc., where 
parameters of the various summary types may also be specified by summary descriptions, e.g., the time duration of 
the video highlight summary. 

[0181] The browsing preferences description scheme may include one or more of the following non-exhaustive list 
of descriptors and descriptions in its description scheme. 

(A) The minimum number of keyframes (Mi nNumOf Keyframes) and the maximum number of keyframes (MaxNu- 
mOfKeyframes) descriptors may be included. These descriptors specify the user's preference for minimum and 
maximum number of frames in a keyframe summary of an audiovisual program. A user can specify these descrip- 
tors according to personal taste, situation, etc., and according to channel bandwidth and terminal resource limita- 
tion. 

(B) The minimum duration (MinSummaryDuration) and the maximum duration (MaxSummaryDuration) descriptors 
may be included. These descriptors specify the user's preference for the length of a highlight summary composed 
of key clips in the video. These descriptors may also, for example, be applied to an audio-only material. A user 
can specify these descriptors according to personal taste, situation, etc., and according to channel bandwidth and 
terminal resource limitations. 

[0182] An example for Summary Preferences description that can be included in usage preferences description is 
provided below. 



</UsagePreferences> 
</BrowsingPreferences> 
<SummaiyPreferences> 

<5uimnaryTypePreference>keyVideoCIips</SummaryTypePreference> 
<MmSummaryDuration><rn>3<ym><3>20 

<MaxSummaryDuration><m>6<7m><s>40</s></MaxSummaryDuration> 
</SummaiyPreferences> 
<'BrowsingPreferences> 
<AJsagePreferences> 



(C) The abstraction fidelity descriptor for universal multimedia access application relates to fidelity of a summary 
abstraction of a program. This preference descriptor may correspond to the variation fidelity descriptor contained 
in the media's variation description specified by Variation Description Scheme in ISO/IEC JTC1/SC29 WG11 
N3246, "MPEG-7 Multimedia Description Schemes, Working Draft v2.0", Noordwijkerhout, March 2000. Alterna- 
tively, the duration and number of keyframes may be defined as the fidelity descriptor. 

(D) The SegmentTheme descriptor(s) may be included, which describes the theme or point of view of a segment, 
e.g., a video or audio clip annotated with its theme or emphasis point. An example summary preference description 
expressing preference for video segments (clips) labeled as "Goal from Spain" and "Replay of Goal from Spain" 
is as follows: 
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</UsagePreferences> 
</BrowsingPrcferences> 
<SummaryPreferences> 

<SummaryTypePreference>KeyVideoCHps</SunijmaryTypePrefereiice> 
<SegmentTheme>Goal from Spain</SegmentTlieme> 
<SegmentThcme>RepIay of goal from Spain</SegmcntTheme> 
</SummaryPrcferences> 
</BrowsingPreferences> 
</UsagePreferences> 

(E) The frame frequency value descriptor may be included to specify the temporal sampling frequency of video 
frames that can be visualized in the browser. The frames provide a visual summary. Depending on the browser, 
they may also provide clickable entry points to the video. The user may click and start playing back the video 
starting from that frame. The frame frequency value descriptor provides similar functionality in terms of shots of 
the video. 

[0183] The source preference description describes the preferred source of multimedia information, such as the 
broadcast or storage medium type (e.g., terrestrial, satellite, DVD), broadcast channel identifier, etc. An example user 
preference description expressing preference for Star Trek episodes available from terrestrial broadcast is as follows. 

<UserIdentifier protection-'true" userName="paul"/> 
<UsagePreferences allowAutomaticUpdate="fake"> 
<FilteringAndSearchPreferences protection="true"> 
<PreferenceValue>5</PreferenceValue> 
<CreationPreferences> 

<Title xml:lang="en" type="origmaT>Star TreWTitle> 
</CreationPreferences> 

<SourcePreferences> 

<PublicationType>TerrestrialBroadcast</PublicationType> 

</SourcePreferences> 
</FilteringAndSearchPreferences> 
</UsagePreferences> 
</UserIdenfifier> 

[0184] The filtering and search preferences description includes at least one of the descriptors of preferred program 
title, genre, language, actor, creator of the program. An example description where user's preference is for news pro- 
grams in English is given below. Such description may be included in user's smart card when he travels to Japan, for 
example. Note that this particular preference description is identified as being specific to Japan and differentiated by 
choosing an appropriate user name. 
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<UserIdentifier protection-'true" userName="paul_in_Japan"/> 
<UsagePreferences alIowAutomaticUpdate="fa!se"> 
<FilteringAndSearchPrefcrences protection- 'true"> 
<PreferenceValue> 1 00</PreferenceVaIue> 

<ClassificationPieferences> 

<Language> 

<LanguageCode>en</LanguageCodc> 

</Language> 

<Genie>News</Genre> 



</ClassificationPreferences> 
</FilteringAndSearchPreferences> 
</UsagePreferences> 
</UserIdentifier> 



25 [0185] The PreferenceValue descriptor provides a technique for prioritizing filtering and search preferences, such 
as the value indicating the degree of user's preference or non-preference. Non-preferences may be expressed by 
assigning a negative (opposite) value to the preference value descriptor. 

[0186] The betterthan and worsethan descriptors may describe which instantiation of preferences the user likes or 
dislikes relatively more compared to another instantiation, where different instantiations are identified using the filtering 
30 and search preference type descriptor. This provides robustness against changes in the preference value descriptor 
automatically, for example, by an agent. 

[0187] The filtering and search preferences description may also contain a description of a preferred review to ex- 
press user's desire for searching for programs that are favorably reviewed by specific individuals. For example, pref- 
erence for movies reviewed by movie critics Siskel and Ebert and found to be "two-thumbs- up" may be described and 
35 included in the filtering and search preferences description. 

[0188] An overview of the entire description scheme is shown in FIG. 29. 

Additional Attributes and Descriptors 

40 [0189] The present inventors came to the additional realization that enhanced functionality for the system may be 
achieved by the incorporation of particular types of information in the descriptions and description schemes. In addition, 
these particular types of information may likewise be incorporated within systems without descriptions and description 
schemes to provide enhanced functionality. An explanation of the additional attributes and descriptors is hereinafter 
provided, followed by a portion of example descriptions. 

45 [0190] After further consideration, in particular with respect to the user interaction with the system, there is a need 
forthe user preference descriptions to include descriptions of the user's preference that relate to how media is recorded 
in the local storage of the system. The media may include, for example, audio and/or video content. For instance, the 
user may want to express a preference forthe quality of the recording technique. The quality may include, for example, 
low quality, medium quality, and high quality. These quality levels refer to the quality of the audio and/or video com- 

50 pression technique that is applied to the media prior to being stored on the local storage. Preferably, the audio and/or 
video content is digital and a digital compression technique is employed. The preference of the recording quality may 
be influenced by the available capacity of the local storage or attributes of the program. Accordingly, the system may 
suggest, select different recording qualities, and/or otherwise record programs at different recording qualities based 
upon the available local storage. 

55 [0191] Current personal recording devices, such as the Replay TV personal video recorder, include a fixed single 
default recording quality that can be chosen by the user by invoking the setup mode. The default setting may be for 
standard, medium or high quality The user manually has to change this default setting to one of the other two options, 
if desired, whenever the user instructs the Replay TV personal recording device to record an upcoming program. In 
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addition, the Replay TV personal recording device records every show that the user is watching in order to support 
live pause and replay functionality. However, the present inventors were surprised to observe that the recording quality 
of the Replay TV personal recording device during watching is fixed and cannot be adjusted. 
[0192] Including the recording quality preferences in the user preference descriptions facilitates certain, previously 

5 unrealized, advantages in a flexible and efficient manner. By using the recording quality preference descriptions in 
connection with the other aspects of the usage preference descriptions, for example with the filtering and search pref- 
erence descriptions, recording preferences may be automatically associated with a particular type of content. For 
example, the system may automatically set to recording quality to "high" for sports programming and "low" for nature 
programming. Sports programming tends to include a lot of fast action which is preferably encoded with high quality. 

10 In contrast, nature programming tends to primarily include little action, which if encoded with low quality, will still look 
acceptable while saving on the storage requirements of the system. 

[0193] The system may select the recording quality of programs (audio and/or video) based on predefined rules for 
particular attributes included within the system or associated with the audio and/or video content. In this manner, the 
system may accommodate for differences in the anticipated content of audio and/or video programs. Thus, the user 

'5 does not have to make manual choices every time when the user desires to record different programs at different 
qualities. The system may have access to information regarding both the user's preferred content and the preferred 
recording qualities for such content, enabling the system to potentially function in an autonomous manner on behalf 
of the user. In this manner, the system may discover and record preferred programs at the user's desired recording 
qualities. Accordingly, the system may permit the user to select the recording quality of programs (audio and/or video) 

20 based on predefined rules for particular attributes included within the system or associated with the audio and/or video 
content. 

[0194] The system may include a learning agent that automatically observes and learns the user's recording prefer- 
ences for particular types of content. If desired, the user may input his preferences into his profile. User preferences 
facilitate multi-user support for personalized recording quality settings for all the users of the device. Also, this likewise 

25 permits manual and/or automatic selection of the recording quality for recording by the system while viewing. In this 
manner, the recording quality of the program being currently watched may be selected based on the content of the 
program or the user's preferences. It is to be understood that the system may include manual and/or automatic selection 
of the recording quality in a system that does not include descriptions and description schemes. 
[0195] The user may desire to express their preferences of safeguard time intervals that precede and follow the 

30 program content in an effort to account for possible shifts in program start and end times. For example, a program may 
be scheduled to be broadcast between 10am and 11am on a particular channel, where the user selects a safeguard 
time interval of 5 minutes before the program starts and a safeguard time interval of 1 0 minutes after the program ends, 
resulting in a recording time from 9:55am to 11 :10am. These safeguard time intervals may be based upon the content 
of the user descriptions and description schemes. This is particularly useful in programming the system in advance to 

35 record one or more programs without the user's explicit intervention to set the safeguard time intervals, which is bur- 
densome to many users. For example, because sports programming typically extends beyond the scheduled time the 
system may include significant additional safeguard time interval at the end of such programming while sitcoms typically 
end when scheduled so little, if any, additional safeguard time interval is normally necessary. It is to be understood that 
the system may include manual and/or automatic selection of the safeguard time intervals in a system that does not 

40 include descriptions and description schemes. 

[0196] The present inventors also came to the realization that consideration should likewise be given to recording 
of audio and/or visual content that is enhanced with hyperlinks, e.g., multiple layers of links to web pages that are 
related to the content, for subsequent, off-line consumption. In this case, the user records not only the audiovisual 
program but also its enhancement data as indicated by the hyperlinks. It is desirable for the user to be able to specify 

45 the maximum number of layers of links that the device should automatically record in order to limit the necessary 
storage requirements in some manner. In addition, a parameter may be selected to indicate the maximum amount of 
local storage to use for such enhancement data which reduce the storage requirements and the time necessary for 
downloading the enhancement data. Selecting the number of layers of links to store assists the user in rationing the 
limited locate storage space in the system. By including the number of link layers as part of the user preferences, it is 

so possible for the users to automatically associate such preferences with media characteristics, such as the genre or 
any other attribute associated with the program. For example, a large number of link layers may be specified for edu- 
cational interactive programs. In addition, the multi-user support provided by the user preference description provides 
personalized specifications for all users of the appliance. It is to be understood that the system may include manual 
and/or automatic selection of the link level or maximum storage permitted in a system that does not include descriptions 

55 and description schemes. 

[0197] As previously described, the system may include device preferences that describe user preferences relating 
to how the user consumes the media using the device. Consuming the media includes utilization of the so-called "trick 
modes" of the device, namely, fast forward and fast reverse. Fast forward and fast reverse include any speed faster 
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than normal playback speed. Digital random access storage media based appliances facilitate a wide range of fast 
forward and fast reverse speeds (e.g. ; 4 times versus 20 times the normal speed) as well as instant skipping by a 
particular amount (e.g., 15 seconds) from one point to another point in the content. Skipping, when allowed, may be 
used to skip commercials or uninteresting parts of programs. After consideration of the capability of fast forwarding, 
fast reverse, and skipping, the present inventors came to the realization that a need exists to provide a technique so 
that the user does not have to select among these multiple parameters every time he or she invokes these device 
functionalities. The Browsing Preference descriptions, previously described, may include the playback preferences 
that in turn include the fast forward, fast reverse, and instant skip preference descriptions. By making such preference 
descriptions a part of the user preference descriptions, users are provided with the opportunity to customize such 
preferences to content characteristics such as the genre or any other attribute associated with a program. In this way, 
the user does not have to make choices each time the function is used. The user may select his or her preferences 
for different programming differentiated on any basis. Alternatively, an agent may automatically learn and populate the 
user's user preference descriptions to provide personalized settings for all the users of the appliance. Furthermore, 
users may edit their user preferences thereby changing their preference on the instant skip duration. Likewise, this 
technique may be similarly used for slow forward and slow reverse, namely, a speed slower than normal playback 
speed. 

[0198] Another set of useful additional information is related to filtering and search preferences, particularly, to pref- 
erences on the format of the preferred content. After consideration the present inventors came to the conclusion that 
there is a need for describing the user's preference for a particular media format. This is especially desirable considering 
that the content may be available in multiple formats, or when the user's device has limited playback capabilities of 
various media formats. The source preferences may include a preferred file format description. Examples of some 
media formats may include, for example, MPEG-1, MPEG-2, MP3, Liquid Audio, Real Player, AC-3, Dolby Digital, 
Wide Screen video format, and Normal Screen video format. It is to be understood that the media format maybe used 
in a system that does not include descriptions and description schemes. 

[0199] While the system described herein includes extensive search, characterization, browsing, and filtering func- 
tions the present inventors came to the realization that yet another suitable program attribute is the date of the creation 
of the content itself. For example, the creation date for a particular movie may be the date during which the original 
movie was created, such as 1930, 1940-1949, or 2000. Also, the creation date for a particular movie may be the date 
on which the movie was re-mastered into its current format, such as DVD. With the creation date data, movies created 
by a particular director between 1 960 and 1 970, or episodes of Star Trek that were created between 1 977 and 1 982 
may be selected. The creation date is particularly useful in the filtering out of reruns of the user's favorite television 
episodes. Otherwise, it may be difficult for the system to distinguish between new (or relatively new) television episodes 
and old (or relatively old) television episodes. The date period may support any desired granularity, such as years, 
decades, hours, months, etc. 

Expression of the Additional Attributes 

[0200] The following descriptions are expressed in XML (Extensible Markup Language), incorporated by reference 
herein. It is to be understood that any other description language may likewise be used. 

[0201] The definition of the user preference description may be as follows. In the following the user Paul decides to 
define a user preference description specifically for sports. Paul's recording quality preference for this particular genre 
is "High". Paul also specifies a longer safeguard period at the end of a sports program (15 minutes) than the safeguard 
at the start (30 seconds), due to possible extensions in the game. Paul expresses preference for single layer of-linked 
content. 
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<UserPreference> 

<UserIdentifier protection-'true" userName- 'pPaul_sports'Y> 
<UsagePreferences allowAutomaticUpdate-' false"> 
<FilteringAndSearchPreferences protection="true"> 
<ClassificationPreferences> 
<Genre>Sports</Genre> 
</ClassificationPreferences> 
</FilteringAndSearchPreferences> 
<DevicePreferences> 

<RecordingPreferences> 

<Quality>High</Quality> 
<SafeguardStartOffset> 

<s>30</s> 
</SafeguardStartOffset> 
<SafeguardEndOffset> 

<m>15</m> 
</SafeguardEndOffset> 
<LinkLayers> 1 </LinkLayers> 
</RecordingPreferences> 
</DevicePreferences> 
</UsagePreferences> 
</UserIdentifier> 
</UserPreference> 

[0202] In the above example description, recording preferences are included in device preferences. In another al- 
ternative embodiment, recording preferences may be included in filtering and search preferences. In this manner, 
recording preferences can be associated with each individual set of filtering and search preferences. It is then possible 
to automatically associate different recording preferences with different preferred title, genre, or any other description 
of preferred content, or their combinations. The inclusion can be explicit, or implicit by making use of links to a separate 
recording preference which may in turn be included in the device preferences. In this case, filtering and search pref- 
erences may also include a binary valued attribute to indicate user's preference for to record or not to record those 
programs that meet user's filtering and search preferences. Recording preferences take effect if user's preference is 
to record. 

[0203] Further, additional program-specific recording related preferences may be included in the filtering and search 
preferences, if the particular instantiation of the filtering and search preferences sufficiently identifies a particular pro- 
gram. In that case, for example, if the preferred program has multiple, regularly scheduled episodes, the following 
preferences may be specified: the number of episodes that is desired to be recorded (e.g., 1 ,2, All), and the number 
of most recent episodes that is desired to be kept in local storage (e.g.; if this number is 1 , a new episode always 
replaces the previous episode.), and whether the system should immediately reserve space for this particular program 
if it is going to be available in the future. 
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Playback preferences on fast forward and fast reverse rates, and instant skip amount may be as follows. 

<UserPreference> 

<UserIdentifier protection-'true" userName- 'paul7> 
<UsagePreferences allow AutomaticUpdate="false"> 
<BrowsingPreferences> 

<PlaybackPreferences> 

<FfwdSpeed>5</FFwdSpeed> 

<FrevSpeed>3 </FRevSpeed> 

<InstantSkip><s>l 5</sx/InstantSkip> 
</PlaybackPreferences> 

</BrowsingPreferences> 

<FilteringAndSearchPreferences protection-' true"> 
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</FilteringAndSearchPreferences> 
<DevicePreferences> 

<RecordingPreferences> 

<Quality>Mediura</Quality> 
<SafeguardStartOffset> 

<m>l</ra> 
</SafeguardBegin> 
<SafeguardEnd> 

<m>l/m> 
<SafeguardEndOffset> 
<LinkLayers>3</LinkLayers> 
</RecordingPreferences> 

</DevicePreferences> 
</UsagePreferences> 
<yUserIdentifier> 
</U serPreference> 

[0205] In the following description, the user specifies preference for MPEG-1 file format. 

<UserPreference> 

<UserIdentifier protection="true" userName- 'paul7> 
<UsagePreferences allow AutomaticUpdate="false"> 

<FilteringAndSearchPreferences protection="true"> 



</CreationPreferences> 

50 

<SourcePreferences> 

<PublicationType>Terrestrial Broadcast</PubIicationType> 

55 
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</SourcePre ferences> 

<MediaFormatPreferences> 

<FileFormat>MPEG- l</FileFormat> 
</MediaFormatPreferences> 



</FilteringAndS^archPreferences> 

</\J sagePreferences> 

</UserIdentifier> 

</UserPreference> 



[0206] The following description includes user's preference on creation date period as a part of c. eation preferences. 
The user has a preference for Star Trek episodes created between 1977 and 1982. In this example, the user has 
preference on the source as terrestrial broadcast. 



<UserPreference> 

<UserIdentifier protection="true" userName- 'pauI"/> 
<UsagePreferencesallowAutomaticUpdate="false"> 

<FilteringAndSearchPreferencesprotection="true"> 
<CreationPreferences> 
<TitIe xml:lang= n en" type="original">StarTrek<yTitIe> 
<DatePeriod> 

<TimePoint> 

<y>1977</y> 
</TimePoint> 
<Duration> 

<y>5</y> 
</Duration> 

</DatePeriod> 
</CreatiooPreferences> 
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<SourcePreferences> 

<PubIicationType>Terrestrial Broadcast</PublicationType> 
</SourcePreferences> 
</FilteringAndSearchPreferences> 
</UsagePreferences> 
</UserIdentifier> 
</U serPreference> 

[0207] It is to be understood that any of the descriptors and description schemes may be located at any portion of 
the system. Merely formatters of illustration, referring to FIG. 30 a recording preferences 541 may include the recording 
quality, safeguard time intervals, and number of linked layers. A playback preferences 543 may include the fast forward, 
fast reverse, and instant skip period. A media format 545 may include the preferred media format and a creation date 
547 may include the preferred creation date period. 

[0208] The terms and expressions that have been employed in the foregoing specification are sued as terms of 
description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding 
equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention 
is defined and limited only by the claims that follow. 



Claims 

1 . A method of using a system with at least one of audio, image, and a video comprising a plurality of frames com- 
prising the step of providing a preferences description (500), describing preferences of a user with respect to the 
use of said at least one of said audio, image, and video, where said description is a description about a recording 
quality at the time of recording at least one of said audio, image, and video on a storage means. 

2. A method of using a system with at least one of audio, image, and a video comprising a plurality of frames com- 
prising the step of providing a preferences description (500), describing preferences of a user with respect to the 
use of said at least one of said audio, image, and video, where said description is a description about at least one 
of safeguard time intervals before a program start time and after a program end time at the time of recording at 
least one of said audio, image, and video on a storage means. 

3. A method of using a system with at least one of audio, image, and a video comprising a plurality of frames com- 
prising the step of providing a preferences description (500), describing preferences of a user with respect to the 
use of said at least one of said audio, image, and video, where said description is a description about a playback 
preferences (543) at the time of playing back at least one of said audio, image, and video. 

4. A method of using a system with at least one of audio, image, and a video comprising a plurality of frames com- 
prising the step of providing a preferences description (500), describing preferences of a user with respect to the 
use of said at least one of said audio, image, and video, where said description is a description about a creation 
date (547) for at least one content of said audio, image, and video. 

5. A method of using a system with at least one of audio, image, and a video comprising a plurality of frames com- 
prising the step of providing a preferences description (500), describing preferences of a user with respect to the 
use of said at least one of said audio, image, and video, where said description is a description about a file format 
(545) at the time of recording at least one of said audio, image, and video on a storage means. 
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